PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents

Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, Dawn Song

arXiv:2505.05849v4cs.CRcs.AI

TL;DR

Indirect prompt injection can redirect LLM agents by placing malicious instructions in external data, while black-box access and diverse agent designs make systematic assessment challenging. AGENTVIGIL uses black-box fuzzing to generate and refine attacks, achieving high benchmark success and transferring to unseen tasks and models.

  • Problem

    Indirect prompt injection can make agents follow malicious instructions embedded in external data, while black-box access and agent diversity complicate systematic risk assessment.

    Source: §1 · pp. 1–2

  • Method

    AGENTVIGIL fuzzes black-box agents by selecting and mutating attack seeds, then scoring variants and iteratively refining its seed corpus.

    Source: §1 · p. 2

  • Results

    71% on AgentDojo for o3-mini and 70% on VWA-adv for GPT-4o, nearly doubling benchmark baselines, with strong transfer to unseen tasks and models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AGENTVIGIL’s attacks extend to real-world agents, where they can redirect web navigation to arbitrary URLs, including malicious sites.

    Source: §1 · p. 2, §6 · p. 9

    The threat model excludes harmful misuse of agents and direct attacks on underlying infrastructure.

    Source: §3 · p. 4

Abstract

from arXiv · show

The strong planning and reasoning capabilities of Large Language Models (LLMs) have fostered the development of agent-based systems capable of leveraging external tools and interacting with increasingly complex environments. However, these powerful features also introduce a critical security risk: indirect prompt injection, a sophisticated attack vector that compromises the core of these agents, the LLM, by manipulating contextual information rather than direct user prompts. In this work, we propose a generic black-box fuzzing framework, AgentVigil, designed to automatically discover and exploit indirect prompt injection vulnerabilities across diverse LLM agents. Our approach starts by constructing a high-quality initial seed corpus, then employs a seed selection algorithm based on Monte Carlo Tree Search (MCTS) to iteratively refine inputs, thereby maximizing the likelihood of uncovering agent weaknesses. We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o, respectively, nearly doubling the performance of baseline attacks. Moreover, AgentVigil exhibits strong transferability across unseen tasks and internal LLMs, as well as promising results against defenses. Beyond benchmark evaluations, we apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.

1. Introduction

Indirect prompt injection lets malicious content in external data redirect an agent from the user’s task, while black-box access, task diversity, and architectural complexity make systematic assessment difficult. AGENTVIGIL addresses this with black-box fuzzing and reports strong benchmark results, transfer to unseen tasks and models, and real-world URL redirection.

Problem

  • Indirect prompt injection manipulates external data retrieved by an agent, potentially making it perform an attacker-chosen task instead of the user’s task.Assessment is difficult because real-world agents are often black boxes and vary in tasks, architectures, tools, and services.Source: §1 · p. 1

Our approach

  • AGENTVIGIL fuzzes black-box agents by selecting and mutating attack seeds, then scoring outputs and adding promising variants to the seed corpus.Its designs include high-quality templates, adaptive coverage-based scoring, MCTS-based seed selection, and customized mutators.Source: §1 · p. 2

Differences from GPTFuzzer

  • Unlike direct-prompt fuzzing, AGENTVIGIL targets indirect injection in multi-step agents, where attackers can influence only external content.It combines black-box reward modeling, adaptive seed selection, semantically guided mutators, and designed initial seeds.Source: §1 · p. 2

Results

  • 71% on AgentDojo for o3-mini and 70% on VWA-adv for GPT-4o nearly double the benchmark baselines; generated attacks also transfer to unseen tasks and models.On unseen tasks, success rates are 65% against o3-mini and 59% against GPT-4o; against unseen Gemini-2-flash-exp, the rate is 67%.Source: §1 · p. 2

2. Related Work

LLM agents combine language models with tools to act in varied environments, but prompt-injection attacks and defenses differ in their access assumptions and applicability. Prior approaches include manual attacks, architecture-specific or white-box methods, direct-prompt fuzzing, and several defense strategies.

LLM agents

  • LLM agents use language models to control non-ML tools for tasks such as web interaction, coding, and personal assistance.Tools can include parsers, browsers, execution environments, and memory or retrieval components.Source: §2 · pp. 2–3

Existing attacks

  • A shopping-review injection can redirect a web agent from finding fingerprint-resistant screen protectors to attacker-chosen URLs, including phishing or malware sites.The figure also describes testing URLs and queries that could expose private data.Source: Figure 1 · p. 3
  • Automated attacks range from white-box methods requiring access to target components to direct-prompt fuzzers that need detailed feedback, limiting their applicability to complex real-world agents.Hand-crafted attacks can be effective but require expertise and often yield inconsistent success.Source: §2 · p. 3

Existing defenses

  • Training-dependent defenses can demand substantial resources and updates, while training-free constraints may detect attacks only after execution or limit functionality.Other defenses rely on human oversight, labeling, or action reversal, which can require significant intervention or impractical assumptions.Source: §2 · p. 3

3. Threat Model

The threat model treats the agent as a black box and the user as benign, while allowing an attacker to manipulate external data to misdirect the agent. It excludes harmful agent misuse and direct attacks on underlying infrastructure.

Blackbox setting of the agent systems

  • The agent is black-box: users and attackers lack access to the underlying LLM’s internals and the agent’s architecture, observing only external behavior.Interactions and observations are limited to the system’s external behavior.Source: §3 · p. 3

User assumptions

  • The user is assumed benign, and the scope excludes harmful misuse of agents and direct attacks on hosting or computational infrastructure.The user’s behavior does not contribute to vulnerabilities or malicious actions.Source: §3 · pp. 3–4

Attacker’s capabilities and goals

  • Attackers may alter external data sources and aim to misdirect the agent toward goals unintended by the user.They can interact with the agent like a legitimate user and test attacks on similar tasks; feedback is whether the attack succeeded.Source: §3 · p. 4

4. Method

AGENTVIGIL generates and iteratively tests indirect-injection prompts across tasks, using seed mutation, coverage-aware scoring, and MCTS-based selection. Its corpus, mutations, and selection process support exploration of prompt variants and prioritization of promising seeds.

4.1. Overview

  • The iterative fuzzing loop tests initial prompt-template seeds across injection tasks, then selects, mutates, and scores variants by attack success and newly compromised tasks.MCTS balances exploration and exploitation as the process refines prompts across agents and tasks.Source: §4.1 · p. 4

4.2. Corpus Collection

  • The initial corpus uses templates from human heuristics, online resources, and prior research, with placeholders and varied strategies such as role-play, delimiters, and obfuscation.The placeholders adapt prompts to the model, user task, and attacker goal.Source: §4.2 · pp. 4–5

4.3. Mutation Design

  • Five helper-LLM mutations—Shorten, Expand, Rephrase, Crossover, and GenerateSimilar—create diverse seed variants using basic strategies without extra heuristics.These mutations can use moderately capable, smaller models such as Llama-3-8B and GPT-4o-mini.Source: §4.3 · p. 5

4.4. Seed Scoring

  • Seed scoring combines attack success rate with a coverage bonus for finding successful attacks on previously failed tasks.The hybrid score values immediate effectiveness and potential to explore new attack patterns across task contexts.Source: §4.4 · p. 5

4.5. Seed Selection

  • MCTS uses UCB1 scores to balance exploitation of high-scoring seeds with exploration of less-visited mutation branches.Visit counts propagate to ancestors, and one or two top-scoring seeds are selected for mutation; UCB1 is preferred over UCT because seed evaluations are computationally expensive.Source: §4.5 · p. 5

5. Evaluation

AGENTVIGIL improves attack success over benchmark baselines on both personal-assistant and web-agent tasks, then evaluates transfer, defenses, ablations, and a real-world case. Results also show that transfer varies across models and that the tested defenses do not consistently prevent attacks.

5. Evaluation

  • Evaluation spans personal-assistant and web-agent benchmarks, using separate fuzzing and test sets and assessing transferability and defenses.AgentDojo evaluates whether both the user task and injection task succeed; VWA-adv evaluation focuses on text triggers.Source: §5 · p. 5, §5.1 · p. 6, §5.2 · p. 7

5.1. Attack Personal Assistant Agents; 5.2. Attacking Web Agents

  • 71% on AgentDojo and 70% on VWA-adv: AGENTVIGIL nearly doubles baseline attack success, reaching 71% versus 38% and 70% versus 36%, respectively.The initial corpora reached 63% on AgentDojo and 54% on VWA-adv before iterative refinement.Source: §5.1 · p. 6, §5.2 · p. 7
  • On AgentDojo, o3-mini-generated adversarial prompts transferred across tasks and to GPT-4o-mini, performed relatively worse on GPT-4o and Claude-3.5-Sonnet, and both attacks were ineffective against Claude-3.5-Sonnet.On VWA-adv, AGENTVIGIL improved absolute success rates by 15% to 40% across models and tasks except Claude-3.5-Sonnet.Source: §5.1 · p. 7, §5.2 · p. 7

5.1. Attack Personal Assistant Agents

  • Against AgentDojo defenses, AGENTVIGIL consistently outperforms the baseline, especially against pi_detector and delimit, although defenses reduce both attacks’ success.Among the tested defenses, delimit is less effective than pi_detector and repeat.Source: §5.1 · p. 7, Table 2 · p. 8

5.2. Attacking Web Agents

  • On VWA-adv, AGENTVIGIL’s success rate falls from 60% without defenses to levels converging with the baseline when defenses are applied.The combined defense does not reduce success further than individual defenses; the authors attribute the degradation likely to complex prompts and limited context.Source: §5.2 · p. 8, Table 3 · p. 8

5.3. Ablation Study

  • Removing the curated initial corpus causes success to plateau after about four iterations, while removing adaptive scoring or MCTS selection slows improvement.Figure 3 compares coverage over fuzzing iterations for AGENTVIGIL and the two ablations.Source: Figure 3 · p. 6, §5.3 · p. 8

5.4. Real-world Case Study

  • A WebArena shopping-site case study shows generated prompts leading an agent to arbitrary URLs, including phishing sites, and to other undesired actions.The experiment used a local copy of the site for ethical considerations; reported actions also include downloading malicious files and sending private information.Source: §5.4 · pp. 8–9

6. Conclusion

AGENTVIGIL combines high-quality prompt templates, adaptive seed scoring, and MCTS-based seed selection to test black-box agents. It achieves high attack success on benchmarks and real-world agents, transfers across unseen tasks and underlying LLMs, and highlights limitations in existing defenses.

6. Conclusion

  • AGENTVIGIL combines high-quality prompt templates, adaptive seed scoring, and MCTS-based seed selection to test black-box agents.The framework is designed to address black-box constraints, architectural complexity, and the wide-ranging functionalities of real-world agents.Source: §6 · p. 9
  • The framework achieves high attack success rates on established benchmarks and real-world agents, with strong transferability across unseen tasks and underlying LLMs.The conclusion describes automated generation and optimization of adversarial prompts.Source: §6 · p. 9
  • AGENTVIGIL's results highlight limitations in existing agent defenses and support further work on security solutions.The authors present the framework as a foundation for advancing understanding of agent-based threats and developing next-generation security solutions.Source: §6 · p. 9

Impact Statement

The work is an offensive testing tool whose results offer insights for agent developers and security researchers. The authors emphasize that testing and defense require ongoing research and proactive updates as threats evolve.

Impact Statement

  • Although the framework is primarily an offensive testing tool, its results offer insights for agent developers and security researchers.The authors say these insights can guide development of more robust defense mechanisms and secure system designs.Source: Impact Statement · p. 9
  • The authors say revealing weaknesses early can help stakeholders protect against malicious manipulations while supporting legitimate, safe use of agent systems.Source: Impact Statement · p. 9
  • The authors caution that no single testing or defense approach is infallible and that evolving threats require ongoing research and proactive updates.Source: Impact Statement · p. 9

A. Detailed Model Checkpoints

The evaluation used specified checkpoints for o3-mini, GPT-4o-mini, GPT-4o, Claude-3.5-Sonnet, and Gemini-2-flash-exp.

A. Detailed Model Checkpoints

B.1. Evaluation on Extra Models

On the latest available o3-mini checkpoint, AGENTVIGIL outperformed the handcrafted attack on the fuzzing and test sets. Additional evaluations report utility scores for several open-source models and use QwQ-32B for further experiments.

B.1. Evaluation on Extra Models

  • 72% on the fuzzing set and 74% on the test set were AGENTVIGIL's success rates on o3-mini-2025-01-31, versus 50% and 53% for the baseline handcrafted attack.Source: §B.1 · p. 12
  • Utility scores were 42% for Llama3.3-70B-Instruct, 54% for Qwen2.5-72B-Instruct, 74% for QwQ-32B, and 79% for o3-mini.The passage defines utility scores as success rates on benign tasks.Source: §B.1 · p. 12
  • The authors conducted further experiments using QwQ-32B after reporting its utility score alongside other models.They state that evaluated open-source models often lag behind in long-context understanding, advanced tool usage, reasoning, and planning needed for challenging agent scenarios.Source: §B.1 · p. 12

B.2. Comparison with Additional Baselines

AgentVigil outperforms state-of-the-art baselines in the AgentDojo comparison, which includes additional attacks and reports results across models and task sets.

B.2. Comparison with Additional Baselines

  • AgentVigil outperforms state-of-the-art baselines, especially in discovering and exploiting indirect prompt-injection vulnerabilities.The comparison includes attacks from OpenPromptInjection and InjecAgent.Source: §B.2 · p. 12
  • The comparison adds OpenPromptInjection and InjecAgent attacks to the baseline set.Source: §B.2 · p. 12
  • Table 4 reports attack success rates on Fuzzing and Test task sets for QwQ-32B and o3-mini.Source: Table 4 · p. 12, §B.2 · p. 12

B.3. Breakdown on Attack Scenarios

AgentVigil outperforms benchmark baselines across the covered AgentDojo suites and VWA-adv attack goals. The section also presents its coverage curve and the seed-scoring and MCTS-selection algorithms.

B.3. Breakdown on Attack Scenarios

  • Across Slack, Workspace, Travel, and Banking, AgentVigil achieves higher success rates than AgentDojo's baseline attacks.Source: §B.3 · p. 12, Table 5 · p. 13
  • On VWA-adv, AgentVigil outperforms the benchmark baseline for both illusioning and goal misdirection.The passage notes this includes challenging goal-misdirection tasks.Source: §B.3 · p. 12, Table 5 · p. 13

B.4. Coverage Curve

C. Algorithms

  • Seed scoring combines attack success rate with a coverage bonus for newly successful task combinations.The bonus is weighted by the coverage factor and normalized by the number of questions.Source: C. Algorithms · p. 14
  • The update algorithm increments ancestor visit counts and adds the new node to the node set.Source: C. Algorithms · p. 14
  • The selection algorithm chooses the highest-UCB node, or the top two nodes when two are requested.Source: C. Algorithms · p. 14
Loading 2505.05849v4…