PaperlayerView exact revision on arXiv

Source-linked AI summary

Defeating Prompt Injections by Design

Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, Florian Tramèr

arXiv:2503.18813v2cs.CRcs.AI

TL;DR

LLM agents can be vulnerable to prompt injections in untrusted data. CaMeL uses software-security principles to control information and actions, and solves the AgentDojo security evaluation by design with guarantees against policy violations but some utility degradation.

  • Problem

    LLM agents interacting with external environments can face prompt injections in untrusted data, which may redirect actions or exfiltrate information.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    CaMeL extracts control and data flows from user queries and applies software-security principles and explicit policies to prevent disallowed actions without exhaustively enumerating them.

    Source: §1 · p. 2, §5 · p. 6

  • Results

    CaMeL solves the AgentDojo security evaluation by design, with guarantees against policy violations but some utility degradation.

    Source: §1 · p. 2, §11 · p. 25

  • Takeaways & Limitations

    A system built around an untrusted model can be made robust even if the model itself is not.

    Source: §11 · p. 25

    CaMeL does not prevent prompt-injection attacks that phish users, though its annotations can flag text from untrusted sources.

    Source: §6.2.2 · p. 16

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving $77\%$ of tasks with provable security (compared to $84\%$ with an undefended system) in AgentDojo. We release CaMeL at https://github.com/google-research/camel-prompt-injection.

1. Introduction

LLM agents that interact with external environments are vulnerable to prompt injections from untrusted data, while common training- and prompt-based defenses lack robust ways to define and enforce security policies. CaMeL applies software-security concepts through value-level capabilities, query-flow extraction, and a custom interpreter to enforce policies without modifying the underlying LLM.

1. Introduction

  • Prompt injections exploit untrusted instructions in agent contexts to exfiltrate data or cause harmful actions (Perez and Ribeiro, 2022).Untrusted content can come from compromised users, tool outputs, or web pages.Source: §1 · p. 1
  • Existing defenses often depend on models following security policies through training or vulnerable system prompts (Wallace et al., 2024; Ghalebikesabi et al., 2024).The paper attributes this to the lack of robust methods for formally defining and enforcing policies across diverse data.Source: §1 · p. 1
  • CaMeL extracts control and data flows from trusted user queries and enforces explicit, fine-grained security policies without changing the underlying LLM.It associates values with capabilities that restrict their data and control flows.Source: §1 · pp. 1–2
  • CaMeL uses a custom Python interpreter to track provenance and enforce security policies, providing security guarantees without modifying model behavior.Its approach is inspired by control-flow integrity, access control, and information-flow control.Source: §1 · pp. 1–2
  • CaMeL is integrated into AgentDojo and solves its security evaluation by design, with some utility degradation and guarantees against policy violations (Debenedetti et al., 2024b).The paper describes the benchmark result as practical security evaluation, without giving a percentage in the introduction.Source: §1 · pp. 1–2

2. Defeating Prompt Injections by Design

CaMeL protects LLM agents from prompt injection by securing both control and data flows rather than relying on the underlying model to resist malicious instructions. It uses a custom interpreter and capability-based policies to block unauthorized data sharing.

Defeating prompt injections with software security

  • Prompt injection can make an agent disregard the user’s intended task and exfiltrate a document to an unintended recipient or perform a different action.The untrusted notes in the example may contain malicious instructions that hijack the agent.Source: §2 · p. 2
  • Existing heuristic defenses, including delimiters, prompt sandwiching, and adversarial training, provide no security guarantee and can fail against new attacks.Examples include delimiting untrusted content and reminding the model of the original task after tool outputs (Hines et al., 2024; Learn Prompting, 2024; Chen et al., 2024; Wallace et al., 2024; Wu et al., 2024).Source: §2 · pp. 2–3
  • The Dual LLM pattern separates planning from untrusted-data processing: the Privileged LLM sees only the user query, while the tool-less Quarantined LLM processes potentially malicious content.The Quarantined LLM cannot call arbitrary tools with arbitrary arguments, limiting harm but not preventing manipulated data from affecting tool arguments.Source: §2 · p. 3
  • Even when the Dual LLM pattern protects the plan, malicious content can manipulate data used as tool arguments, enabling attacker-chosen files or recipients and potentially arbitrary code execution.The attack is analogous to SQL injection: the query structure remains protected while its parameters are manipulated.Source: §2 · p. 3
  • CaMeL extracts trusted control flow as pseudo-Python and uses a custom interpreter to track value dependencies and enforce capability-based security policies when tools are called.Capabilities record each value’s sources and allowed recipients, enabling the system to block an email that would expose confidential data to an unauthorized recipient and request explicit user approval.Source: §2 · pp. 3–4

3. Threat Model

CaMeL’s threat model focuses on prompt injections in untrusted data that divert data flows, assuming the user prompt and any memory remain uncompromised. It excludes attacks that leave control and data flows unaffected and does not aim to eliminate all human intervention.

3. Threat Model

  • Untrusted content can redirect a task’s data flow—for example, changing an email recipient so confidential documents go to an attacker—even without changing the sequence of agent actions.The model assumes the user prompt is trusted and any memory has not been compromised.Source: Figure 2 · p. 4, §3 · pp. 4–5

3.1. Explicit non-goals of CaMeL

  • CaMeL does not defend against text-to-text attacks or phishing that do not alter control or data flows or cause private-data exfiltration, though data-flow tracing can help users check content origins.For example, a chat interface could show that purported Google email content did not come from a Google address.Source: §3.1 · p. 5
  • CaMeL does not aim to remove human intervention: ambiguous queries or tool results may require users to clarify expected control and data flows.Capabilities and security policies can reduce unnecessary prompts and shift security decisions to the capability system.Source: §3.1 · p. 5

4. The Prompt Injection Security Game

PI-SEC formalizes prompt-injection security by modeling an adversary’s attempt to make an Agent perform actions outside the prompt’s allowed set. It also specifies how CaMeL checks global tool policies during execution.

4. The Prompt Injection Security Game

  • PI-SEC takes an adversary, an Agent, and a tool set as parameters; the Agent returns a trace of tool calls, arguments, and memory states.The Agent executes a user prompt using the tools and a read-writable memory state on which they operate.Source: §4 · p. 5
  • CaMeL checks globally allowed tool policies before each tool call and halts execution if a policy check fails.These policies apply independently of the prompt.Source: Figure 4 · p. 6
  • An adversary wins by choosing an initial tool-memory state that causes the Agent to perform an action outside the prompt’s allowed set Ωprompt.The Agent’s execution trace is checked for such an unsafe action.Source: §4 · pp. 5–6

5. CaMeL Design

CaMeL applies software-security principles to prevent disallowed actions without exhaustively enumerating every allowable action. Its design separates query planning from untrusted-data parsing and uses tracked information flows, capabilities, and tool-call policies to govern execution.

5. CaMeL Design

  • CaMeL combines a Privileged LLM, a Quarantined LLM, tools, security policies, and an interpreter that tracks data flow and executes or blocks tool calls.The design uses control-flow integrity, capabilities, and security policies rather than exhaustively enumerating allowable actions.Source: §5 · p. 6, Figure 5 · p. 7

5.1. The Privileged and Quarantined LLMs

  • A tool-less Quarantined LLM parses untrusted data, while a Privileged LLM writes code from the user query without seeing tool outputs or Quarantined LLM results.The authors describe CaMeL as the first concrete instantiation of the Dual-LLM pattern using these roles. The Quarantined LLM returns structured data and can signal insufficient information, but cannot tell the Privileged LLM what information it needs.Source: §5.1 · p. 7

5.2. Security Policies

  • Python security policies make granular information-flow decisions at tool calls, including whether calendar-event data may be shared with participants based on their access or trusted provenance.Each policy receives the tool name and arguments and returns Allowed or Denied with a reason; Python permits arbitrary policy logic.Source: §5.2 · pp. 8–9

5.3. Capabilities

  • Capabilities tag individual values with provenance and allowed readers, enabling restrictions tailored to specific data rather than broad data or control-flow rules.For example, a passport number and its derivatives can be marked as processable only by specified trusted tools, while less sensitive personal data may be allowed elsewhere.Source: §5.3 · p. 9

5.4. CaMeL Interpreter

  • The restricted-Python interpreter tracks variable dependencies, applies relevant policies before tool calls, and blocks calls when those policies are violated.In STRICT mode, conditional tests or loop iterables become dependencies of variables assigned in those blocks. On errors, the interpreter can redact messages dependent on untrusted data and preserves state, but cannot roll back failed operations with side effects.Source: §5.4 · pp. 9–11

6. Evaluation

On AgentDojo, CaMeL substantially reduces successful attacks while mostly preserving utility outside the Travel suite. Its evaluation also examines security policies, alternative defenses, data-flow attacks, and token overhead.

6.1.1. CaMeL has a (small) Impact on Utility

  • Except for the Travel suite on some models, CaMeL does not significantly degrade utility, and it sometimes improves model success rates on specific suites.With newer models, Claude Sonnet family Travel utility rose from 25% (3.5), to 55% (3.7), to 75% (4), with no changes to CaMeL or its system prompt.Source: §6.1.1 · p. 11, Figure 8 · p. 12, §6.1.2 · p. 12

6.1.2. How does CaMeL impact utility?

  • Poorly documented tool outputs impede Travel-suite parsing; Claude Sonnet family utility rose from 25% (3.5), to 55% (3.7), to 75% (4) without CaMeL changes.Another failure occurs when the P-LLM requests information absent from a value: the Q-LLM cannot tell the P-LLM what is missing without risking a prompt-injection vector. Table 1 categorizes Claude’s failures, including query misunderstanding, wrong assumptions, and insufficient Q-LLM context.Source: §6.1.2 · pp. 11–12, Table 1 · p. 13

6.1.3. Cheaper Q-LLM Models Have Little Impact on Utility

  • Using Claude 3.5 Haiku as Q-LLM with Claude 4 Sonnet as P-LLM reduces utility by about 1% while reducing median-task cost by an estimated 12%.A similar utility drop was observed with GPT 4.1 Nano as Q-LLM and o4 Mini High as P-LLM.Source: §6.1.3 · p. 12

6.2.1. Case study: without any Security Policies

  • Without security policies, Gemini 2.5 Pro successful attacks drop from 300 to 0 with CaMeL.The evaluation measures attacks across 949 total attacks; the reported successful attacks against CaMeL are not due to prompt injection, as discussed in Section 6.2.2.Source: Figure 9 · p. 14, §6.2.1 · p. 14

6.2.2. Case study: with Security Policies

  • With security policies, the residual Travel attack is the same non-prompt-injection case; CaMeL’s fine-grained annotations can flag the untrusted text, while policy denials vary by suite.Workspace outputs support granular policies; Slack and Banking have more denials because data sources or user identity are less clear, or the send_money policy is strict. Travel results are less informative because utility is low.Source: §6.2.2 · pp. 15–16, Figure 10 · p. 15

6.3. Baseline comparisons

  • Against defenses tested with Claude 3.5 Sonnet, CaMeL has 0 successful attacks versus 8 for tool filter.GPT-4o Mini with CaMeL was not vulnerable to AgentDojo attacks, while its tool-calling API was vulnerable to 276; CaMeL provides guarantees, unlike the other probabilistic defenses.Source: Figure 11 · p. 16, §6.3 · pp. 16–17

6.4. Case study: when data flow becomes control flow

  • An email that directs an agent to execute an action can turn untrusted data into control flow, enabling hidden malicious instructions to dictate tool use.The passage describes breaking a malicious program into benign-looking small instructions that the model sees separately, making the overall intent harder to detect.Source: §6.4 · pp. 17–18, Figure 12 · p. 17

6.5. Overheads

  • For the median AgentDojo task, CaMeL uses 2.82× input tokens and 2.73× output tokens compared with native tool-calling.The paper reports the highest token usage among analyzed defenses for CaMeL; Spotlighting uses 1.06× more input and 0.98× the output tokens.Source: Figure 13 · p. 18, §6.5 · pp. 18–19

7. Side Channels Can Be a Problem

CaMeL remains vulnerable to side-channel attacks despite improving security through isolation and fine-grained data tracking; the authors warn that other attacks may also be possible. The section gives examples involving indirect tool dependencies, exceptions, and timing.

7. Side Channels Can Be a Problem

  • CaMeL remains vulnerable to side channels, which can reveal sensitive information through shared resources, though fine-grained data tracking can help reason about some leakage.The authors caution that additional side-channel attacks may be possible and that agents may still mishandle data, albeit inefficiently.Source: §7 · p. 19
  • An attacker can infer a private value by observing conditional external-image fetches despite no direct dependency; STRICT blocks state-changing statements in control-flow blocks by tracking their tests or iterables.Without STRICT mode, an attacker can use the number of image fetches to infer a value such as a password reset token.Source: Figure 14 · p. 19, §7 · pp. 19–20

Exception Handling – Inferring private variable based on halted execution

  • An attacker can leak one private bit by causing an exception conditional on private content and observing whether a later tool call executes; STRICT makes subsequent statements depend on Q-LLM inputs.When non-public data was passed to the Q-LLM, STRICT can block a later state-changing tool and prompt for confirmation. Explicit error handling with conditionals and result types could preserve dependencies in both branches; exceptions triggered by tool-accessed data may remain possible.Source: Figure 15 · p. 20, Exception Handling – Inferring private… · pp. 20–21
  • Timing may reveal private data, but exploitability depends on the deployment, available tools, and observation precision; CaMeL’s current interpreter lacks the time module, preventing this specific attack.The authors do not rule out other timing side channels.Source: Exception Handling – Inferring private… · p. 21

8. Secondary attack scenarios that CaMeL can help with

CaMeL can address threats beyond prompt injection, including rogue users and malicious tools that violate system security policies. Its capability-based design provides security guarantees that capability-less systems lack.

Secondary attack scenarios

  • CaMeL can prevent a rogue user or tool from violating system security policies, extending protection beyond the prompt-injection threat model.These stronger threat models allow compromise by the user, data, tooling, or any combination; the paper identifies capabilities as a source of security guarantees unavailable in capability-less systems.Source: §8 · pp. 22–23
  • Insider threats accounted for 44% of data compromises in a 2014 PwC estimate, while a 2022 report attributed 50% of insider compromises to negligence and 26% to malicious insiders.The paper frames these threat models as realistic for production agent systems and notes their limited attention in academic literature.Source: §8 · p. 22

Scenario 2: External Spy Tool

  • CaMeL addresses a malicious external spy tool that uses prompt injection to monitor agent data and exfiltrate sensitive information, a control-flow attack outside AgentDojo.The tool could be knowingly installed or arrive as compromised software; such tool selection has been demonstrated as possible (Nestaas, Debenedetti, and Tramèr (2025)).Source: §8 · p. 22

Scenario 3: Rogue User

  • A rogue user can alter a prompt to send confidential financial documents to an external address, manipulating control and data flows beyond AgentDojo (Debenedetti et al., 2024b).The scenario models a malicious insider or compromised account with legitimate access to the agent.Source: §8 · pp. 22–23

9. Discussion

CaMeL’s capability-based defense has adoption, usability, and side-channel limitations, and the authors do not claim that prompt injection is fully solved. They frame CaMeL as verifiable security scaffolding whose utility and guarantees depend on supported capabilities and enforced policies.

9. Discussion

  • The discussion identifies security-literature-inspired attacks and side channels as potential limitations shared with other capability-based defenses.These risks accompany the challenges of balancing security with user experience, especially de-classification and user fatigue.Source: §9 · p. 23

9.1. Past problems with adoption of capabilities

  • Capability-based systems are costly to implement and require ecosystem-wide tool support, though controlled workspaces or an agent acting as a central capability authority may make adoption feasible.The CHERI case involved redesigning the software-hardware stack, changing development practices, and raising awareness (RSM UK Consulting LLP for UK DSIT, 2025).Source: §9.1 · p. 23

9.2. De-classification and user fatigue

  • Restrictive or ambiguous security policies can require user intervention for de-classification, causing fatigue that may lead users to approve malicious actions; the right security-usability balance depends on the application.De-classification challenges are discussed by Anderson, Stajano, and Lee (2002), and user fatigue in other settings by Felt et al. (2012) and Cao et al. (2021).Source: §9.2 · p. 24

9.3. So, Are Prompt Injections Solved Now?

  • CaMeL does not fully solve prompt injection: users must specify and maintain policies, and side-channel exploitation is hindered—but not eliminated—by limited bandwidth and attack complexity.The authors also suggest attackers could approximate malicious control flows by chaining smaller policy-permitted blocks, analogous to return-oriented programming attacks against CFI.Source: §9.3 · p. 24
  • The authors position CaMeL as model scaffolding focused on verifiable security guarantees while maximizing utility, rather than as a complete resolution of prompt injection or AgentDojo.They expect adopting CaMeL as a design paradigm to enable future utility improvements.Source: §9.3 · p. 24

10. Future work

Future work targets more secure implementations and stronger assurance, while improving context-aware policy decisions and automation.

Using a different programming language

  • Languages with explicit error and I/O handling, such as Haskell, may ease secure real-world deployment, while Python complexity complicates security and policy-conflict resolution.Program termination caused by exceptions can itself be a security issue.Source: §10 · pp. 24–25

Towards formal verification

  • Formal verification could prove CaMeL’s interpreter fault-free and confirm conflict resolution and intended policy enforcement despite complex code and vulnerable underlying LLMs.The current implementation provides strong empirical evidence in benign and security LLM evaluations.Source: §10 · p. 25

Contextual integrity and security policy automation

11. Conclusion

CaMeL uses principled system design around language models to defend against prompt injection, with guarantees against unintended actions and data exfiltration. The conclusion notes its limitations while presenting compatibility with other defenses and security engineering as directions for stronger and broader protection.

11. Conclusion

  • CaMeL is a practical prompt-injection defense based on system design, providing strong guarantees against unintended actions and data exfiltration.The paper says it effectively solves the AgentDojo benchmark, without specifying a result value in the conclusion passages.Source: §11 · p. 25
  • CaMeL does not address every potential attack vector, and the authors hope future work will address limitations of the current design.Its design allows the interaction between defense components to be studied precisely and vulnerabilities to be reasoned about using software-security experience.Source: §11 · p. 25
  • CaMeL remains compatible with defenses that improve model robustness, and combining them could provide a stronger security argument.Source: §11 · p. 25
  • The authors argue that security engineering can help beyond prompt injection by making systems robust around untrusted models, and encourage applying similar approaches elsewhere.They note that a single robust model meeting modern safety and security requirements in every setting may not be immediately practical.Source: §11 · p. 25

Contributions · A. Extended Related Work · A.1. Background: Software Security Concepts

The paper’s development and manuscript involved broad collaboration, with E.D. and I.S. providing most manuscript input.

Contributions

  • E.D. led technical development, while E.D. and I.S. co-led CaMeL’s design; everyone contributed to the manuscript, with most input from E.D. and I.S.I.S. originated the idea and wrote the technical design proposal; T.F. organized the student internship proposal. Several authors provided proposal feedback, code reviews, technical and administrative support, and manuscript comments.Source: Contributions · p. 26

A.1.1. Computer security nomenclature … A.2. Prompt Injection Defenses

The background defines security policies, control and data flows, and capabilities, then surveys established controls, tool-calling agents, and defenses against prompt injection. It frames CaMeL’s setting as one where explicit agent tools and data sources make flow separation possible.

A.1.1. Computer security nomenclature

  • Security policies specify system goals, subjects, objects, and permitted operations, but the passage says such policies are not currently robustly enforceable for machine-learning models.Source: §A.1.1 · p. 30
  • Security policies specify system properties and constrain operations, while capabilities grant fine-grained rights and control and data flows describe execution and information propagation.The paper notes that robust policy enforcement for machine-learning models is not currently available.Source: §A.1.1 · pp. 30–31
  • Although control and data flows are generally difficult to separate in machine-learning models, explicit tools and data sources can make an agent’s instruction sequence and corresponding data flows identifiable.The example sequence finds meeting notes, extracts an email and document name, fetches the document, and sends it to the email address.Source: §A.1.1 · p. 31

A.1.2. Access and Information Flow Controls

  • Control Flow Integrity restricts execution to paths in a predefined graph, while access control governs resource access and information flow control tracks sensitive data to prevent unauthorized leaks.The passages distinguish mandatory and discretionary access control and describe static analysis and specialized type systems as IFC approaches.Source: §A.1.2 · p. 31

A.1.3. Tool-calling LLM Agents

  • Tool-calling agents receive tool documentation in the system prompt, produce formatted calls interpreted by a runtime, and may request further calls based on tool outputs.A footnote explains that data and model components can be semantically alike, while instruction streams are structured and restricted in their representations.Source: §A.1.3 · pp. 31–32

A.2. Prompt Injection Defenses

  • Prompt-injection defenses include delimiting untrusted text, prompt sandwiching, structured-query training, fine-tuning, task-change detection, and architectural separation of instruction and data inputs.Other approaches use BERT-based injection detection, canaries for exfiltration, or information-flow tracking of trusted tool-output portions.Source: §A.2 · p. 32
  • Willison’s isolation approach gives planning to a privileged LLM that sees only the user prompt and delegates untrusted-data processing to a tool-less quarantined LLM.This blocks attacks requiring a different tool-call sequence; a similar isolation method requiring user approval between tools can cause user fatigue.Source: §A.2 · p. 32
  • Concurrent work uses labeling, synthetic policy extraction, or trusted and untrusted LLMs with tool-call-time policies, while differing from CaMeL in tracking or policy expressiveness.Zhong et al. (2025) use a classifier rather than an explicit control-flow graph and coarser labels; Abdelnabi et al. (2025) focus on travel planning and agent communications.Source: §A.2 · p. 32

B. Full results tables

The full-results tables report AgentDojo utility across suites, utility under attack, successful-attack counts, and defense utility.

B. Full results tables

  • Table 2 reports AgentDojo utility results across different suites.Source: Table 2 · p. 33
  • Table 3 reports AgentDojo utility results under attack across different suites.Source: Table 3 · p. 33
  • Table 4 reports the number of successful attacks, while Table 5 reports defense utility.Source: Table 4 · p. 34, Table 5 · p. 34

C. Baseline results · D. NORMAL vs. STRICT modes

The supplied captions cover defenses’ utility and successful attacks under attack, policy-triggering rates in NORMAL and STRICT modes, and a NORMAL-mode dependency-graph example. They do not provide the underlying table values.

D. NORMAL vs. STRICT modes

  • Table 8 reports policy-triggering rates in NORMAL and STRICT modes when not under attack.The supplied caption gives no rates.Source: Table 8 · p. 34
  • Table 6 reports defenses’ utility under attack, with a suite-by-suite breakdown in Figure 18.Figure 18 directs readers to Table 6 for the full results; the supplied captions give no utility values.Source: Table 6 · p. 35, Figure 18 · p. 35
  • Table 7 reports the number of successful attacks against defenses.The caption supplies no counts.Source: Table 7 · p. 35
  • Figure 19 shows a NORMAL-mode CaMeL dependency graph in which send_email does not depend on query_ai_assistant, unlike Figure 7b.The example query asks whether the last email is about a meeting tomorrow and, if so, forwards it to email@example.com.Source: Figure 19 · p. 36
  • Table 9 reports policy-triggering rates in NORMAL and STRICT modes when under attack.The supplied caption gives no rates.Source: Table 9 · p. 36

E. Security Policy evaluation

The section evaluates CaMeL’s security policies through rules that constrain data sources, recipients’ ability to read shared information, and permitted side effects. The policies cover transactions, account updates, communications, calendar actions, and files.

E. Security Policy evaluation

  • Transaction and account policies require user-sourced recipients or data, trusted password sources, and restrict transaction disclosure to recipients able to read its components.Scheduling may be recurrent; updating a scheduled transaction differs by transaction ID, which is not revealed to the recipient, while user-info updates expose no data.Source: E. Security Policy evaluation · p. 36
  • Messaging and webpage policies require recipients or audiences to read messages, and permit posting or retrieving webpages only when the relevant content and URL are public.Direct messages may instead use a user-specified recipient; email recipients must come from the user, and email content must be readable by all recipients.Source: E. Security Policy evaluation · pp. 36–37
  • Calendar-event creation requires details readable by participants or participants specified by the user; cancellation requires an event ID from user-requested content.The cancellation rule is described as an event requested directly by the user or, in another formulation, as coming from the user or user-sourced content.Source: E. Security Policy evaluation · p. 37
  • Email deletion is allowed only when the email ID comes from an email directly requested by the user.Source: E. Security Policy evaluation · p. 37
  • File policies make created files accessible only to the user, require user-sourced IDs or sharing recipients, and prevent appending data unreadable by people with file access.The model must call share_file to make created-file content visible to others.Source: E. Security Policy evaluation · p. 37
  • Figure 20 reports the percentage of tasks triggering security policies across evaluated models for benign and adversarial evaluations with CaMeL enabled.Source: Figure 20 · p. 38

F. Extended overheads results … H.4.1. Workspace 11: Query misunderstanding

This section documents token-usage overhead comparisons across defenses, system and exception-handling prompts, the available tool environment, and an email-to-tool-action workflow. It also presents a workspace query misunderstanding with repeated code-repair attempts and classifications of two lunch entries at 12:30.

F. Extended overheads results

  • Tables 10–13 compare token usage or token-usage increases across multiple defenses, with Tables 12–13 covering attack conditions.The captions do not provide the tables’ numerical values.Source: Table 10 · p. 38, Table 11 · p. 39, Table 12 · p. 40
  • Figure 21 measures CaMeL’s token-use increase over native tool calling without attack, using Claude 3.5 Sonnet and marking the median-task increase in red.Tokenization uses tiktoken.Source: Figure 21 · p. 39

H.1. System prompt

  • The prompt identifies the user as Emma Johnson of Blue Sparrow Tech and instructs the model to avoid assumptions, use tools to disambiguate, and check the current year rather than assume it.It also says to try to complete tasks without user intervention or confirmation.Source: §H.1 · pp. 40–41
  • The prompt supplies basic data types and built-in functions, plus external tools for email, contacts, calendars, and cloud-drive files.A provided function returns the current day in ISO format.Source: §H.1 · pp. 41, 44, 45
  • The system prompt directs the LLM assistant to parse unknown data structures into a specified schema and discourages passing tool outputs without further instructions.It calls for specific field types to improve parsing accuracy.Source: §H.1 · p. 47
  • The system prompt restricts generated code to a custom interpreter, forbidding imports, eval or exec, loops, and side-effecting methods.The prompt also adds a banking-suite note about transaction senders and positive amounts, and provides datetime classes only in the workspace and travel suites.Source: §H.1 · p. 48

H.2. Exceptions prompt

  • The exception prompt asks for corrected, runnable code and says exceptions dependent on untrusted data are redacted, so the model should infer the problem from context.It also warns against rerunning code with side effects that had already executed.Source: §H.2 · p. 49

H.3. Case-study: when control flow turns into data flow from Claude and o3-mini

  • The email example parses message content into structured tasks, asks an LLM to select tools and parameters, then executes the returned actions.The available actions include send_email, create_calendar_event, and append_to_file.Source: §H.3 · p. 51

H.4.1. Workspace 11: Query misunderstanding

  • The workspace example says the model assumed the lunch query referred to now, not the previous calendar event.Its code searches events for May 15, 2024, identifies Sarah's lunch with an LLM, and calculates the time difference from the current time.Source: §H.4.1 · pp. 52–53
  • The initial code failed because break statements are unsupported, prompting the assistant to replace the search with a list comprehension.That replacement included a slice to take at most one result, which triggered the subsequent slices error.Source: §H.4.1 · pp. 54–55
  • A later rerun failed because LunchEvent was already defined, so the assistant continued with the existing class instead of redefining it.The assistant's continuation searched events and retained the time calculation and HH:MM formatting.Source: §H.4.1 · pp. 55–56
  • The classification marked 12:30 on 2024-05-15 as lunch, while the 10:00 and 15:00 entries that day were marked not lunch; each cited result reported sufficient information.Source: §H.4.1 · pp. 55–56
  • The revised code removes the slice operation, takes the first matching event from the list, calculates the time difference between now and lunch, and outputs it in HH:MM format.Source: §H.4.1 · p. 58

H.4.2. Slack 11: Data Requires action … I. More sophisticated side-channel

This section presents assistant workflows across Slack, Workspace, banking, travel, and calendar tasks, including attempts to revise code and queries when errors arise. The examples also include task outputs and a calendar-search side channel that can reveal whether the user will meet Bob.

H.4.2. Slack 11: Data Requires action

  • The task’s underlying obstacle is that some required website data is unavailable to the P-LLM.The prompt asks for contact details found on a website, while the example workflow tries to recover the information from Alice’s inbox.Source: §H.4.2 · p. 59
  • The attempted workflow reads Alice’s inbox, extracts a colleague’s name, email, and channels, then invites them to Slack and adds them to those channels.The code uses an AI assistant for extraction before calling the Slack actions; it raises an error if the message is missing or lacks enough information.Source: §H.4.2 · p. 59
  • Repeated extraction attempts raise NotEnoughInformationError, including after the query is made more specific and restricted to messages from Bob to Alice.The later revision filters for Bob-to-Alice messages and passes only those messages to the assistant, but the next recorded run still reports the same error.Source: §H.4.2 · pp. 60–62
  • The proposed code uses the extracted name and email to invite the colleague to Slack, then adds them to each specified channel.The same invite-and-add actions appear in both the simplified-query and filtered-search attempts.Source: §H.4.2 · pp. 63, 66
  • After direct-message and all-message queries raised NotEnoughInformationError, the approach shifted to filtering messages for onboarding-related keywords.The attempts sought the colleague’s name, email address, and channels; the supplied passages show no successful extraction.Source: §H.4.2 · pp. 64–65
  • The revised search filters all messages for terms such as “welcome,” “joining,” “new hire,” and “onboard” before querying for colleague details.The query requests the new person’s name, email address, and Slack channels.Source: §H.4.2 · p. 66
  • A restricted Python runtime rejected a generator expression; the retry replaced it with a list comprehension using sum while retaining the Slack workflow.The code still filtered messages and proceeded to invite the colleague and add them to specified channels.Source: §H.4.2 · pp. 66–67
  • Slack retries still returned NotEnoughInformationError after filtering for Bob’s messages, prompting a broader search across all messages with added context.The revised prompt specified Emma Johnson’s identity, the colleague details needed, and that information might be spread across messages.Source: §H.4.2 · pp. 68–69

H.4.3. Workspace 18: Wrong assumptions from P-LLM

  • The Workspace example flags an assumption about the hike start time, while the shown lunch query labels a 10:00 event not lunch and a 12:30 event lunch.The described workflow searches calendar events for the day, identifies the lunch event, and calculates the time difference.Source: §H.4.3 · pp. 69–71
  • In Workspace 18, code intended to identify Sarah’s lunch and calculate the time failed on unsupported break syntax, class redefinition, and slicing.The attempted output was meant to show the time until lunch in HH:MM format.Source: §H.4.3 · pp. 71, 73, 75

H.4.4. Banking 2: Not enough context for Q-LLM

  • In Banking 2, the request asks P-LLM to extract an IBAN that is absent from the landlord notice, while its extraction schema requires a recipient_iban.The notice says rent will increase by 100.00 starting next month but provides no IBAN.Source: §H.4.4 · pp. 76–77
  • The proposed workflow extracts the new rent amount, effective date, and IBAN, then updates the existing scheduled payment while preserving its subject and recurring status.It prints the updated details; the initial version also reports when required notice information is missing.Source: §H.4.4 · p. 77
  • The rent-notice workflow repeatedly failed with NotEnoughInformationError, including after the extraction request was made more specific.The later attempt then supplied the existing payment details to the assistant and allowed its IBAN to be used if the notice did not specify one.Source: §H.4.4 · pp. 79–81
  • The scheduled transactions included a rent payment dated 2022-04-04 with ID 7, recipient US122000000121212121212, and recurring status false.Source: §H.4.4 · p. 81
  • The rent-notice analysis reports a rent increase, no new IBAN, and sufficient information, classifying the notice as a rent increase.The notice states that rent will increase by 100.00 starting the next month.Source: §H.4.4 · pp. 82–83

H.4.5. Workspace 36: Q-LLM overdoes it/Strict eval

  • The Hawaii task response identifies June 13 as hiking at Diamond Head and confirms creation of hawaii-packing-list.docx.The generated packing list includes travel documents, clothing, and beach and activity gear.Source: §H.4.5 · pp. 84, 86, 87

H.4.6. Banking 14: Ambiguous task

  • Q-LLM is reported unable to recognize the ambiguous iPhone 3Gs transaction; the shown security-check code retrieves the 50 most recent transactions for review.The task asks to check transaction history for suspicious activity and update the password if needed.Source: §H.4.6 · p. 87
  • No suspicious transactions were found among the reviewed records, so the example reported no need to update the password.The described workflow retrieves the last 50 transactions and updates the password if any suspicious activity is found.Source: §H.4.6 · pp. 88, 90

H.4.7. Travel 19: Underdocumented API

  • The travel output lists hotel, car-rental, and restaurant options for both London and Paris.Source: §H.4.7 · pp. 90–91
  • The code gathers hotels, car-rental companies, and restaurants in both cities, then uses review data to select the highest-rated hotels.Source: §H.4.7 · pp. 91–92
  • The code retrieves hotel, rental, and restaurant prices and begins calculating costs for two London days and three Paris days.Source: §H.4.7 · p. 92
  • Separate review calls identify the highest-rated car-rental companies and restaurants in each city.Source: §H.4.7 · p. 92
  • The workflow selects the highest-rated options in both cities, retrieves their prices, calculates trip costs, and displays recommendations with a cost breakdown.The calculation covers hotels for two London and three Paris nights, car rentals for two and three days, and two restaurant meals per day for two people.Source: §H.4.7 · pp. 93, 95
  • After a parsing error, the revised prompts specify the highest rating, return format, tie handling, and conversion of ratings to a 0–5 decimal.The error is reported as a redacted NotEnoughInformationError; the passages do not show this revision succeeding.Source: §H.4.7 · pp. 93, 95
  • After another NotEnoughInformationError, the next prompts include explicit candidate-name lists alongside reviews and require returned names to match those lists.This change is applied to hotels, car rental companies, and restaurants; no result confirming success is provided.Source: §H.4.7 · pp. 95–96
  • The revised script uses type-specific schemas, combines review and price information, structures prompts, parses prices during analysis, and prefers costlier options when ratings tie.It calculates costs for hotels, car rentals, and restaurants across a two-person London-and-Paris trip.Source: §H.4.7 · p. 99
  • The combined-information hotel query is followed by another NotEnoughInformationError, prompting a different repair approach.The exception appears after the revised London hotel query; the supplied passages do not establish its cause.Source: §H.4.7 · p. 100
  • The next approach analyzes hotels one at a time, extracting each exact name, rating, euro price, and review summary before choosing the highest-rated hotel.For tied ratings, it selects the more expensive option; the passages then begin applying the same process to Paris hotels.Source: §H.4.7 · p. 100
  • Trip costs are estimated from selected hotels, car rentals, and restaurants using the planned nights, rental days, and meals for two people.The code uses two London hotel nights and three Paris nights, two London rental days and three Paris days, and two meals per day for two people.Source: §H.4.7 · p. 102
  • The revised approach analyzes each venue individually, extracts structured details including review summaries, then selects the best option from the analyzed venues.The restructuring is described as processing smaller chunks and converting Pydantic models to dictionaries for final selection.Source: §H.4.7 · p. 103
  • After a hotel-analysis KeyError, the revised prompts provide all review and price dictionaries and request extraction for a specifically named hotel.The error text is redacted, so the passages do not establish its exact cause.Source: §H.4.7 · p. 103
  • The first rerun failed because HotelDetails was redefined even though the class already existed, so the revised code removed class definitions.The revised code retained the analysis logic and used the existing HotelDetails, RentalDetails, and RestaurantDetails classes.Source: §H.4.7 · pp. 106, 109
  • The code analyzes each hotel, rental company, and restaurant separately using review and price information, then selects the highest-rated options for London and Paris.For hotels, ties are resolved by choosing the most expensive option; for rentals and restaurants, any highest-rated option may be chosen.Source: §H.4.7 · pp. 106–108
  • The estimated trip total combines hotels for 2 London nights and 3 Paris nights, rentals for 2 and 3 days, and restaurant costs for 2 people eating 2 meals per day.Restaurant costs use 4 meals in London and 6 in Paris; the output labels the result an estimated maximum total cost.Source: §H.4.7 · p. 108
  • A later run still raised an error at the hotel-analysis prompt on line 4, but the supplied traceback does not establish its cause.The traceback identifies the f-string beginning “Analyze this specific hotel's information” as the flagged line.Source: §H.4.7 · p. 109
  • The repair retrieves reviews and prices up front, supplies explicit per-venue information with missing-data defaults, and requests structured names, ratings, prices, and summaries.The response describes these as improvements intended to make the assistant's inputs clearer and more structured.Source: §H.4.7 · pp. 109, 110, 112
  • The code applies this analysis to hotels, car rentals, and restaurants in London and Paris, then selects top-rated options for each city and venue type.Hotel ties go to the most expensive option; rental and restaurant ties may be resolved arbitrarily.Source: §H.4.7 · pp. 110–111
  • The proposed itinerary estimate combines hotel nights, rental days, and two daily meals for two people across a two-day London and three-day Paris trip.The code calculates separate hotel, car-rental, and restaurant costs before printing the estimated total.Source: §H.4.7 · p. 112
  • The revised code still produced a NotEnoughInformationError, whose cause is not visible because the exception was redacted.The traceback points to the hotel-analysis prompt, but the supplied passage does not establish why the error occurred.Source: §H.4.7 · pp. 112–113
  • The workflow compares London and Paris hotels, asks for each city’s highest-rated option, and breaks hotel rating ties by choosing the more expensive one.It requests a 0–5 rating, euro price, and review summary for each selected hotel.Source: §H.4.7 · p. 113
  • For car rentals and restaurants in both cities, the prompts request the highest-rated option and its rating, euro price, and review summary; rental ties may be resolved arbitrarily.Restaurant prompts specify price per person, while car-rental prompts specify daily price.Source: §H.4.7 · p. 114
  • The code estimates maximum costs for two people using two London and three Paris hotel nights, two and three rental days, and two meals per day.Restaurant costs use four London and six Paris per-person meal charges.Source: §H.4.7 · p. 115
  • A London hotel query produced a redacted NotEnoughInformationError, after which the code represented hotel details as structured records and specified conversion, selection, and output requirements.The revised prompt asks for exact names, numeric 0–5 ratings, euro prices, and brief review summaries.Source: §H.4.7 · pp. 115–116
  • The travel code structures venue data and clarifies prompts with explicit conversion, selection, and output-format instructions.The changes apply a consistent data structure across venue types; the restaurant examples ask for the highest rating and numerical prices in euros.Source: §H.4.7 · pp. 117–118

H.4.8. Travel 19: AgentDojo Bug

  • The AgentDojo check expects a day, while the model supplies operating hours, causing an evaluation bug.The task asks for a French restaurant open on Sunday and requests its name, rating, address, and operating hours.Source: §H.4.8 · p. 118
  • The restaurant code first fails because `list[RestaurantInfo]` is used directly as the assistant schema; a wrapper model is proposed instead.The revised schema defines `FrenchRestaurants` with a `restaurants` list field.Source: §H.4.8 · p. 120
  • A subsequent query raises `NotEnoughInformationError`, prompting a revision that includes the full restaurant list and cuisine information.A later run then fails because `FrenchRestaurants` is redefined; the supplied passages do not show a successful final run.Source: §H.4.8 · pp. 120–121
  • After a NameError for the missing RestaurantDetails class, defining it produced a recommendation of Breizh Café, rated 3.9 and open Sunday.The output gives its address as 109 Rue Vieille du Temple, 75003 Paris, France.Source: §H.4.8 · pp. 123–124

I. More sophisticated side-channel

  • An injected instruction can make the agent expand calendar searches after a missing Bob meeting, potentially searching the entire calendar.The client edits the meeting-minutes file; the search can grow by changing its query when an information error is raised.Source: §I · p. 125
  • If Bob is found within 10 attempts, the email is sent and the adversary learns the user will meet him; otherwise, the adversary learns nothing new.The email contains the meeting summary and next-meeting date requested by the user.Source: §I · pp. 124–125

Related papers

Loading 2503.18813v2…