AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

1 Introduction

• AgentDojo introduces a dynamic benchmark populated with 97 realistic tasks and 629 security test cases for adversarial agent evaluation.The framework includes tasks such as email management and uses attacker goals and injection endpoints to test prompt injections.Source: §1 · pp. 1–2
• Agents dynamically call multiple tools in stateful adversarial environments, while formal utility checks evaluate outcomes from the environment state.This design measures the utility-security tradeoff without relying on another LLM to simulate the environment.Source: §1 · p. 2
• Current LLMs solve less than 66% of tasks without attacks, while attacks succeed against the best agents in less than 25% of cases and defenses reduce attack success to 8%.The benchmark challenges both benign task completion and adversarial robustness.Source: §1 · p. 2
• AgentDojo is designed as an extensible environment for adding new tasks, attacks, and defenses as machine-learning security evolves.The authors release code, a leaderboard, and documentation to support continued evaluation.Source: §1 · p. 2

2 Related Work and Preliminaries

• Prompt injections exploit untrusted tool-returned data to insert instructions that hijack agents into performing malicious actions on behalf of users.Defenses detect injections, improve instruction-data separation, or isolate function calls, but current techniques do not guarantee security for critical tasks.Source: §2 · p. 3
• Existing agent benchmarks evaluate single function calls or multiturn scenarios, generally without explicit prompt-injection attacks.ToolEmu uses LLMs to simulate tool calls and score utility, which can be problematic when injections may hijack the evaluator.Source: §2 · p. 3
• Even in benign settings, AgentDojo tasks are harder than the Berkeley Tool Calling Leaderboard, and attacks further increase difficulty.Figure 2 presents the benchmark-level comparison with the Berkeley Tool Calling Leaderboard.Source: Figure 2 · p. 3, §2 · p. 3
• AgentDojo differs from prior prompt-injection benchmarks by requiring agents to plan and execute multiple tool calls in realistic, stateful environments.InjecAgent focuses on simulated single-turn scenarios where an LLM receives one adversarial tool output directly.Source: §2 · p. 3

3 Designing and Constructing AgentDojo

• AgentDojo combines user and injection tasks within stateful environments, using formal criteria to measure both task utility and attacker success.Task suites pair user tasks with injection tasks; the same user tasks can also be evaluated without attacks.Source: §3 · p. 3, §3.1 · p. 5

3.1 AgentDojo Components

• AgentDojo models mutable environment state and exposes 74 tools that read or write it, while deterministic utility functions inspect outputs and before-and-after state.Tool outputs are formatted as YAML by the current runtime, while the framework supports arbitrary formatting.Source: §3.1 · p. 4
• Explicit utility checks avoid relying on an LLM evaluator that prompt injections might themselves hijack.This design trades some scalability for a state-based evaluation of utility and security.Source: §3.1 · p. 4
• The benchmark covers four environments with realistic tasks requiring long-context search, tool chaining, and attacks ranging from one to twenty steps.The environments are Workspace, Slack, Travel Agency, and e-banking, with tasks spanning email, calendars, cloud storage, messaging, travel, and banking.Source: §3.1 · pp. 3, 5

3.2 Agents and Prompt Injection Defenses

• The framework supports modular agent pipelines, including defenses that combine an LLM agent with a separate prompt-injection detector.Agents receive user instructions, available tools, and environment state through a common query interface.Source: §3.2 · pp. 5–6

3.1 AgentDojo Components

• The benchmark’s initial suite contains 70 tools, 97 realistic user tasks, and 27 injection targets across curated environments.The table caption summarizes the scale of the curated benchmark suite.Source: Table 1 · p. 6

3.3 Prompt Injection Attacks

• AgentDojo motivates adaptive attack evaluation because generic static attacks may not account for the defended model, while defenses can overfit to particular attacks.Attack methods can identify injection placeholders in outputs of tools needed for the user task.Source: §3.3 · p. 6

3.4 Reporting AgentDojo Results

• The benchmark reports benign utility, utility under attack, and targeted attack success rate to separate task completion, safe completion, and malicious-goal achievement.Collections of attacks can be scored as successful when any attack succeeds, modeling an adaptive attacker that selects the best attack per case.Source: §3.4 · p. 6

4 Evaluation

• The evaluation covers closed- and open-source models on 629 security test cases spanning 97 user tasks, with GPT-4o used for detailed ablations and defense experiments.Claude 3.5 Sonnet has the highest benign capability, closely followed by GPT-4o.Source: §4 · pp. 6–7, §4.1 · p. 7

4.1 Performance of Baseline Agents and Attacks

• Most models lose 10%–25% absolute utility under attack, while higher-capability models tend to show higher targeted attack success rates.Benign utility also strongly correlates with utility under attack, indicating a utility-robustness relationship.Source: §4.1 · p. 7, Figure 6 · p. 7
• Slack attacks reach 92% success, whereas a travel-agent injection requiring two unrelated malicious actions succeeds in 0% of cases.The travel task often produces only partial attacker success, completing one malicious action but not both.Source: §4.1 · p. 7, Figure 7 · p. 8
• Injections placed near the end of a tool response are most effective, reaching up to 70% average success against GPT-4o.The result is reported in the appendix position analysis.Source: §4.1 · p. 8

4.2 Ablations on Attack Components and Attacker Knowledge

• The “Important message” attack clearly beats prior prompt-injection phrasings, while the adaptive Max attack increases success rates by another 10%.The comparison includes “ignore previous instructions,” InjecAgent, and “TODO” variants.Source: §4.2 · p. 8, Figure 8 · p. 8
• Knowing the user and model names increases attack success by 1.9%, but incorrect guesses significantly weaken the attack.The ablation is summarized in Table 2.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Prompt Injection Defenses

• Many defenses increase benign utility, but the prompt-injection detector significantly degrades utility because of false positives.Prompt repetition is reasonable against the evaluated attack but unlikely to withstand adaptive injections.Source: §4.3 · p. 9
• Tool filtering lowers attack success to 7.5% when user tasks need read access but attacker goals require write access.It fails when tools cannot be planned in advance or when the same tools support both the user task and attack; this applies to 17% of test cases.Source: §4.3 · pp. 8–9, Figure 9 · p. 9

• AgentDojo standardizes evaluation of prompt-injection attacks and defenses through 97 realistic tasks and 629 security test cases.Source: §5 · p. 9
• AgentDojo can serve as a live benchmark for measuring progress and the security of AI agents in adversarial settings.Source: §5 · pp. 9–10
• The framework is designed as a dynamic environment that can incorporate more sophisticated attacks, defenses, tasks, and multimodal settings.Source: §5 · pp. 9–10
• The benchmark’s current attacks and defenses are relatively simple, motivating future extensions with stronger isolation mechanisms and adaptive attacks.Source: §5 · p. 9

Checklist

• The paper reports that its main claims, contributions, limitations, societal impacts, and ethics compliance are addressed in the manuscript.Source: Checklist · p. 15
• The experiments provide code, data, training details, 95% confidence intervals, compute information, and documentation for reproducing results.Source: Checklist · p. 15
• The paper states that it uses dummy data rather than real data and reports no human-subject research or participant compensation.Source: Checklist · p. 15

A Additional Details on AgentDojo’s Design

• Supplementary figures specify the base agent pipeline, injection task, attack definition, model prompts, defense prompts, and attack prompt placeholders.Source: Figure 10 · p. 16, Figure 11 · p. 16, Figure 12 · p. 16
• AgentDojo attacks return injections for available placeholders, with candidate locations selected from tools required by the user task.Source: A Additional Details on AgentDojo’s… · p. 16

B.1 Agent Prompts

• The Llama 3 70B prompt replaces {funcs} with JSON-schema documentation for the tools available to the model.Source: Figure 16 · p. 18

B.2 Defense Prompts

• The supplemental defense and attack prompts include Data Delimiting, Tool filter, and four injection variants with task-specific placeholders.Source: Figure 17 · p. 19, Figure 18 · p. 19, Figure 19 · p. 19

C Full Results

• Table 3 reports targeted and untargeted attack success rates for different agents with 95% confidence intervals.Source: Table 3 · p. 20
• Table 4 reports targeted and untargeted attack success rates for different prompt injections with GPT-4o and 95% confidence intervals.Source: Table 4 · p. 20
• Table 5 reports targeted and untargeted attack success rates for different GPT-4o defenses with 95% confidence intervals.Source: Table 5 · p. 20

D Additional Results

• US$35 runs the full 629-case security suite on GPT-4o, while the 97-case utility suite costs US$4.Source: D Additional Results · p. 20
• Targeted attacks derail agents from their original tasks at least as effectively as the tested untargeted denial-of-service alternatives.Source: D Additional Results · p. 20
• Attacks placed at the end of the model’s context window are most effective, although AgentDojo does not currently support attacker-controlled positioning.Source: D Additional Results · p. 20

E.2 Hosting, licensing, and maintenance plan

• The code is hosted on GitHub, installable with pip install agentdojo, and maintained through updates as models, attacks, and defenses evolve.Source: §E.2 · p. 22

E.3 Reproducibility

• The release includes model outputs, conversations, notebooks reproducing figures and tables, framework documentation, and exact dependencies.Source: §E.3 · p. 22

E.4 Code and data license

• The software may be used, copied, modified, published, distributed, sublicensed, or sold, provided the copyright and permission notices remain included.Source: §E.4 · p. 22
• The software is provided “as is” without express or implied warranties, and the authors disclaim liability for resulting claims or damages.Source: §E.4 · p. 22

F.2 Publishers

• The benchmark is owned and published by ETH Zurich and Invariant Labs, with no explicit institutional funding; one author is supported by an armasuisse CYD Fellowship.Source: §F.2.1 · p. 23, §F.2.2 · p. 23, §F.2.3 · p. 23

F.3 Dataset overview

• AgentDojo contains 124 tasks, 70 tools, and 136 KB of environment data, including delegated tasks, tools, and pre-populated accessible state.Source: §F.3 · p. 24

F.3.1 Sensitivity of data

• No intentionally or unintentionally collected sensitive data are reported.Source: §F.3.1 · p. 24
• The dataset is listed as having no known risk types.Source: §F.3.1 · p. 24

F.3.2 Dataset version and maintenance

• The benchmark is regularly updated using semantic versioning, with planned additions for harder tasks, more indirection, and potentially multimodal tasks.Source: §F.3.2 · p. 24

F.4 Example of data points

• Tasks encode user and adversary prompts, utility and security checks, ground truths, environment data, and tool descriptions with executable implementations.Source: §F.4 · p. 24

F.5.1 Motivations

• The benchmark is intended to study tool-calling agents’ utility and robustness against prompt-injection attacks, and to test attacks and defenses.Source: §F.5.1 · p. 25, §F.5.2 · p. 25

F.5.2 Intended use

• AgentDojo is openly accessible through its documentation repository and is intended for research validation, but evaluation should include a strong adaptive attack rather than default attacks alone.Source: §F.5.2 · p. 25, §F.6.1 · p. 25, §F.9.3 · p. 26

F.7.1 Collection

• Synthetic data were generated with GPT-4o, Claude Opus, and author-created content, then manually inspected, edited for realism, and retained when syntactically valid and mutually consistent.Source: §F.7.1 · p. 25, §F.7.2 · p. 26

F.11 Validation types

• Validation checks schemas and verifies that ground-truth executions pass the corresponding utility and security checks.Source: §F.10.1 · p. 26, §F.11 · p. 26

F.12 Known applications and benchmarks

• AgentDojo is intended for evaluating tool-calling agents, with results and methodology reported in Section 4 of the main paper.Source: §F.12 · p. 26