PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming

Yanting Wang, Chenlong Yin, Runpeng Geng, Jinyuan Jia

arXiv:2608.05108v1cs.CR

TL;DR

Prompt injection red-teaming is costly and existing reinforcement-learning methods generalize poorly to new target LLMs. PIMiner builds reusable attack strategies through hierarchical memory and transfers them across target and attacker LLMs, achieving strong benchmark performance.

  • Problem

    Prompt injection red-teaming must efficiently evaluate agent vulnerabilities and generate useful attack data despite costly repeated target-agent interactions and context consumption.

  • Method

    PIMiner converts attack history into reusable knowledge through a strategy library, router, iterative attack module, and hierarchical memory.

  • Results

    PIMiner’s learned attack strategies exhibit strong transferability across target and attacker LLMs on IPIArena and AgentDojo.

  • Takeaways & Limitations

    PIMiner’s interpretable strategy library can support agentic-system auditing and training-data generation for stronger defenses.

Abstract

from arXiv · show

Prompt injection poses significant security risks to LLM agents. Efficient and effective red-teaming is therefore critical, both for evaluating these risks and for collecting training data to improve defenses. Existing state-of-the-art prompt injection red-teaming methods primarily rely on reinforcement learning (RL), producing attacker models that often generalize poorly to new target LLMs. In this work, we develop PIMiner, an agentic system for prompt injection red-teaming. During training, PIMiner is trained on a sequence of (dataset, target model) pairs and builds a strategy library from scratch. At test time, the learned strategy library can be directly transferred to a previously unseen target LLM without additional training. PIMiner requires only a small number of queries to a target agent (e.g., 10) per test sample. Experimental results demonstrate that PIMiner achieves strong performance. On IPIArena, it attains a 76.2% ASR against Gemini-2.5-Pro, 61.9% ASR against GPT-5.1, and 42.9% ASR against Claude-Sonnet-4.5. On AgentDojo, it achieves an 86.7% ASR against Gemini-2.5-Pro, 53.3% ASR against GPT-5.1, and 40.0% ASR against Claude-Sonnet-4.5.

1 Introduction

Prompt injection threatens deployed LLM agents, making effective red-teaming important for security evaluation and training-data collection. PIMiner addresses limitations of existing RL- and search-based methods by organizing attack history into reusable strategies for transferable red-teaming.

  • Motivation: LLM agents can use tools, retrieve information, and act autonomously, but are highly vulnerable to prompt injection from untrusted context sources.Examples of untrusted sources include webpages and retrieved content.
  • Motivation: Red-teaming systematically evaluates prompt injection vulnerabilities and produces successful attacks that can support high-quality training datasets.Model developers assess agent robustness before deployment, while collected attacks can improve defenses through training.
  • Prior Methods: RL-based methods achieve strong attack performance but typically require many target-agent interactions, whereas search-based methods optimize each sample independently and are less effective.The introduction attributes this gap to search methods’ lack of accumulated, reusable attack knowledge.
  • PIMiner: PIMiner transforms attack history into reusable knowledge through hierarchical memory containing a strategy library, router, iterative attack module, and experience digestor.The router selects relevant strategies for each sample given an agent dataset and target model.
  • Evaluation: PIMiner is evaluated on IPIArena and AgentDojo against frontier LLMs, with learned attack strategies exhibiting transferability across target and attacker LLMs.The evaluated targets include GPT-5, GPT-5.1, Claude-Haiku-4.5, Claude-Sonnet-4.5, and Claude-Opus-4.5.

2 Related Work

Prior prompt-injection red-teaming methods include RL-based optimization and strategy-based search, while agentic-system design may be manual or automated. PIMiner addresses limitations of fixed strategy libraries by discovering and refining strategies from attack experiences across tasks and target models, despite high evaluation costs.

  • Prompt-Injection Red-Teaming: Prompt-injection red-teaming methods mainly use RL-based attacker optimization or strategy-based search.RL approaches optimize attacker LLMs over many training samples, whereas strategy-based methods guide search with predefined libraries.
  • Prompt-Injection Red-Teaming: Predefined strategy libraries are not dynamically updated for new tasks, limiting their adaptation.PIMiner instead discovers strategies from scratch and continuously refines its library using attack experiences across user tasks, injected tasks, and target models.
  • Agentic Systems: Agentic systems can be designed manually with human-crafted heuristics or automatically through agent optimization frameworks.Although automatic agent design has shown promise on standard agentic tasks, applying it to prompt-injection red-teaming is difficult because evaluation is costly.

3 Problem Formulation

PIMiner is formulated as an agentic prompt-injection red-teaming system that updates a long-term strategy library across dataset–target-model pairs and transfers it to unseen test targets. The formulation assumes grey-box access during training and black-box access during testing.

  • Training setup: During training, the agent observes a sequence of agent-dataset and target-LLM pairs, which may reuse the same dataset with different target models.One example is an AgentDojo training split paired with GPT-5-nano.
  • Training setup: For each dataset–model pair, the system generates attacks on samples and updates its long-term memory from the target model’s output feedback.The memory evolves from S_t to S_t+1 after collecting feedback from M_t.
  • Test-time operation: At test time, the agent uses its evolved memory S_T to generate prompt-injection attacks against a previously specified target model on test-dataset samples.The test pair is (D_test, M_test), and the transferred memory is used without the training-time sequence of updates described above.
  • Attacker access: The attacker has grey-box access to the target agent during training and black-box access during testing.During training, the attacker can observe the target agent’s output at each agent step, including the step where malicious text is injected.

4 Design of PIMiner

PIMiner addresses the effectiveness–cost trade-off in prompt injection red-teaming with hierarchical memory and selective strategy retrieval. Its pipeline routes samples, iteratively optimizes attacks using multiple memory sources, and digests experiences to update a transferable strategy library.

  • Memory Design: PIMiner organizes memory into a long-term strategy library, intra-dataset memory, and intra-sample memory, while selectively loading information to manage inference costs.These levels provide transferable strategies, current dataset–model-pair experience, and feedback from the current sample.
  • Strategy Library: The strategy library begins with _TEMPLATE.md and stores each strategy’s model and task scopes, injection template, examples, failure conditions, and metadata.The library serves as PIMiner’s long-term memory and grows through experience digestion.
  • Strategy Router: The router selects a small Top-K subset of relevant strategies for each sample and retains a cold-start template when no existing strategy applies.It matches routing summaries against the sample and target model, loading only the selected candidates into the attacker context.
  • Iterative Attack Module: The iterative attack module refines each injected prompt until success or the maximum iteration budget, using routed strategies, intra-sample feedback, and intra-dataset experience.Intra-sample feedback includes prior prompts, target trajectories, outcomes, and attacker analyses, enabling diagnosis-driven refinement.
  • Experience Digester: After samples reach terminal states, the experience digester analyzes their trajectories and updates the strategy library with successful attack examples or broader-scope coverage.This completes the pipeline of routing, iterative optimization, and strategy-library updating.

5 Evaluation

PIMiner is evaluated across diverse agent benchmarks, target LLMs, and attack baselines using ASR@N, with strong results and evidence that its memory and transferable strategy library improve performance. Its attacks achieve high success rates across model families, remain competitive with RL-based methods, and transfer beyond Claude attacker LLMs.

  • Evaluation setup: The evaluation uses IPIArena and AgentDojo, with 20 training samples and held-out test samples from each benchmark.IPIArena contains 41 samples; AgentDojo spans domains including workspace management, banking, travel, and Slack.
  • Evaluation setup: PIMiner is compared with static, search-based, and RL-based attacks, reporting ASR@10 unless otherwise specified.The search-based baselines include TAP, PAIR, and Strategy; RL-based baselines include Vanilla GRPO, RL-Hammer, and PISmith.
  • Main results: 76.2% and 86.7% ASR@10 are achieved on IPIArena and AgentDojo, respectively, against Gemini-2.5-Pro.PIMiner is evaluated on nine frontier LLMs spanning the DeepSeek, Gemini, GPT, and Claude families.
  • Main results: 42.9% ASR@10 is achieved on IPIArena against Claude-Sonnet-4.5, alongside 28.6% ASR@1.Human attackers achieve only 1.0% ASR@1 on average in the cited IPIArena results, while Claude-Opus-4.5 is substantially more robust.
  • Baseline comparison: 1.0 ASR is achieved on all three InjecAgent targets, matching RL-Hammer and PISmith and substantially outperforming static and conventional search-based attacks.On AgentDojo, PIMiner achieves 0.53 versus PISmith’s 0.38 on GPT-5-nano, 0.73 versus 0.78 on GPT-4o-mini, and 0.63 versus 0.81 on GPT-4.1-nano.
  • Ablation studies: 19.8% and 17.8% average-ASR improvements over the vanilla iterative attacker result from combining long-term strategy-library and intra-dataset memory.Removing either memory level consistently degrades performance on held-out target LLMs.
  • Ablation studies: 43% and 61% reductions in attacker input length are obtained by the router on Claude-Haiku-4.5 and Claude-Sonnet-4.6, respectively.The router reduces inference cost while retaining substantially shorter inputs.
  • Cross-family transfer: The learned strategy library improves attack performance when integrated into PAIR with attacker LLMs outside the Claude family.A router selects the three most relevant strategies for attacker models including Gemini, GPT, and DeepSeek variants.

6 Discussion and Limitations

PIMiner’s pipeline is compatible with different backbone LLMs, but the work primarily uses Claude Code models to reduce inference costs. Training incurs approximately $20 in additional target-model API costs, compared with substantially more target queries for RL-based methods.

  • Model Compatibility and Cost: PIMiner’s pipeline is general and compatible with different backbone LLMs, but primarily uses Claude Code models to reduce inference costs.The passage notes that Appendix G provides a detailed cost analysis.
  • Model Compatibility and Cost: $20 is the approximate additional target-model API cost during training when users provide their own Claude Code subscriptions.The attacker, router, and digester agents are powered by these subscriptions.
  • Model Compatibility and Cost: 10,000 queries are typical for RL-based methods during training, which can cost more than $100.This is contrasted with PIMiner’s approximately $20 additional target-model API cost.

7 Conclusion

PIMiner is an agentic prompt injection red-teaming system that converts past attack experience into reusable, human-readable attack knowledge. Its hierarchical memory combines three memory components with a router that keeps attacker context compact.

  • PIMiner converts past attack experience into reusable, human-readable attack knowledge for prompt injection red teaming.
  • The system uses a hierarchical memory mechanism combining a long-term strategy library, intra-dataset memory, and intra-sample memory.
  • A router keeps the attacker’s context compact while coordinating PIMiner’s memory components.

Ethical Considerations

PIMiner’s automated red-teaming capabilities carry dual-use risks, but the authors frame proactive vulnerability identification as essential to secure LLM development. Experiments were conducted only on public benchmarks and models in controlled, sandboxed environments without real systems, users, or private data.

  • Dual-use risk: PIMiner can generate effective adversarial prompts, creating dual-use risks.The system is designed to evaluate LLM agents’ robustness against prompt injection.
  • Responsible development: Proactively identifying and disclosing vulnerabilities is presented as essential for responsible secure-LLM development.The authors also characterize automated red-teaming as a core part of model-deployment pipelines.
  • Controlled evaluation: All experiments used publicly available benchmarks and models in controlled, sandboxed environments without real-world systems, users, or private data.Evaluations covered IPIArena, AgentDojo, and InjecAgent; reproduced attacks were tied to benchmark samples rather than real systems.

A Prompt Injection Defenses · B Complete Structure for the Strategy File · C Router Agent’s Prompt Template

The section distinguishes external prompt-injection defenses from internal alignment, describes PIMiner’s reusable strategy-file structure, and specifies how a router selects candidate strategies for each test case. The router matches target-model and test-case characteristics, prioritizes concrete strategies, and returns up to three candidates in a fixed format.

  • A Prompt Injection Defenses: External defenses include detection-based defenses, input sanitization mechanisms, and security-policy-based approaches operating outside the target LLM.
  • A Prompt Injection Defenses: Detection and sanitization can face adaptive evasion and inference overhead, while policy-based defenses require defined tool interfaces and accurately specified security policies.
  • A Prompt Injection Defenses: PIMiner focuses on red-teaming the internal alignment of state-of-the-art language models rather than evaluating external defense mechanisms.
  • B Complete Structure for the Strategy File: Each strategy file is a structured Markdown memory item recording applicability, distinctions from related strategies, instantiation guidance, successes, and failure conditions.
  • B Complete Structure for the Strategy File: The router uses target-LLM scope, task scope, and in-context examples, whereas the attacker and digester use the full strategy file.
  • C Router Agent’s Prompt Template: The router selects strategies by matching target-model strength and test-case surface, including suite, tool-output format, and malicious goal.
  • C Router Agent’s Prompt Template: The router returns the TOP 3 best-fit strategy ids, best first, and uses the COLD-START FALLBACK only when no concrete strategy plausibly fits.

D Attacker Agent Details

PIMiner’s attacker agent separates persistent role instructions from per-iteration attack context, enabling iterative refinement from target feedback and explicit cross-sample memory. It also uses rolling parallelization to maintain a fixed pool of active attackers within each dataset–model pair.

  • Prompt structure: The attacker prompt has a static role component defining the red-team identity, objectives, and expected analysis-plus-injection outputs.The static component evolves at a different timescale from the dynamic context.
  • Prompt structure: The per-iteration context includes the target task, injection location, and complete prior outcomes, and is refreshed every iteration for strategy refinement.The attacker is instructed to identify the latest failure mode and address it specifically.
  • Runtime execution: At runtime, the persistent role prompt and CLAUDE.md load at session launch, while an orchestration program supplies the changing context as tool output during the attack loop.This ordering ensures the agent receives the persistent instructions before per-iteration information.
  • Cross-sample memory: A fixed 20K-character memory block by default summarizes earlier samples, allowing reuse of successful patterns and avoidance of failed ones for the current dataset–model pair.Curate summarizes terminal samples using the task, injection goal, clipped context, final injected prompt, and closing analysis.
  • Parallel execution: A default pool of 5 attacker agents runs in rolling parallelization, launching a new pending sample whenever an active sample terminates.The pool remains full until all samples are completed, improving wall-clock efficiency.

E Digester Agent Details

The digester is PIMiner’s post-training learning component: it analyzes successful and failed attacks after each run and updates a reusable strategy library for future routing and attacking. It classifies attacks by mechanism, preserves evidence-backed examples and refinements, creates strategies for novel mechanisms, and incorporates miss patterns as failure conditions.

  • Learning and Library Updates: After each training run, the digester analyzes successful and failed prompt injections and updates the strategy library used by both the router and attacker.It is implemented as a Claude Code agent and invoked once after each completed training run.
  • Mechanism-Based Classification: The digester classifies attacks by underlying mechanism rather than surface wording or application domain, distinguishing fits-as-is, fits-but-widens, and novel cases.A known mechanism on a materially broader application surface widens an existing strategy; only an undescribed mechanism is novel.
  • Strategy Refinement: For matched attacks, it appends full prompt-injection examples and may refine strategy scope, applicability conditions, templates, and other documented sections using cited evidence.Novel mechanisms receive new strategy files, while examples preserve the polluted context, winning injection, and resulting tool call verbatim.
  • Failure Learning: Miss-pattern analysis runs even on zero-hit runs, downgrading overbroad scope or adding structural failure conditions when predicted hits fail.Failure-condition additions cite the relevant run directory.

F A Qualitative Analysis of the Learned Strategies

PIMiner discovers seven distinct attack strategies, including benchmark-specific variants, with a highly uneven distribution of examples across strategies and target LLMs. The qualitative analyses show complementary mechanisms, while some strategies remain highly model-specific and fail under target-specific filters or incompatible slot conditions.

  • Strategy library: PIMiner discovers seven distinct attack strategies, with three strategies automatically receiving separate AgentDojo- and IPIArena-tailored variants.The resulting library contains benchmark-specific adaptations for Authenticated-Principal Voice Forge, Fabricated Procedure Gate, and Forged Chat Turn.
  • Strategy distribution: 50 examples belong to Fabricated Procedure Gate and 29 to Forged Chat Turn, together accounting for approximately 80% of all examples.The distribution across strategies and target models is highly uneven, indicating that relative strategy effectiveness varies by model.
  • Strategy distribution: Several learned strategies remain highly model-specific, having been discovered and refined against a single target model.This model specificity contrasts with the uneven cross-model distribution of in-context examples.
  • Learned mechanisms: Fabricated Procedure Gate frames malicious actions as mandatory procedures or system-mandated steps, using authoritative prose or integration-style diagnostics rather than conversational role markers.Its example presents a registrar-required drop-then-add sequence to induce a course-registration tool call.
  • Learned mechanisms: Forged Chat Turn inserts textual role delimiters into polluted tool output to create an apparent user turn containing consent, the malicious action, and canonical tool arguments.The strategy fails when target models do not tokenize role markers as turn boundaries and in several target-specific task settings.
  • Learned mechanisms: Neutral source-code documentation can induce canonical actions without authority, persona, role markers, or imperative framing, but fails when classifiers, slot type, or file consistency block that interpretation.The docstring example caused the model to emit a health-check command by treating the injected text as missed helper documentation.

G Cost Analysis

PIMiner’s training costs approximately $20 in out-of-pocket API usage with a Claude Code subscription, while test-time cost is driven mainly by up to 10 target-agent attack iterations per sample. Compared with RL-based red-teaming, PIMiner reduces query costs and avoids attacker policies’ limited transferability across target models.

  • Test-phase cost: 10 is the test-time maximum number of target-agent attack iterations per sample in the experiments.Assuming access to a Claude Code plan, per-sample cost is primarily determined by querying the target agent.
  • Training-phase cost: $20 is the estimated out-of-pocket API cost for PIMiner’s training phase with a Claude Code subscription.The estimate excludes Claude Code subscription usage.
  • Comparison with RL-based methods: 10,000 queries are typically required by RL-based methods during training, costing more than $100 for models such as GPT-5.These methods also often produce attacker policies with limited transferability across target models.
Loading 2608.05108v1…