PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Inject: Automated Prompt Injection via Reinforcement Learning

Xin Chen, Jie Zhang, Florian Tramèr

arXiv:2602.05746v2cs.LGcs.AI

TL;DR

Prompt injection is hard to automate because it requires specific tool calls, yet current attacks rely heavily on hand-crafted prompts and jailbreak optimizers target generic compliance. AutoInject learns adversarial suffixes with comparison-based reinforcement learning, and outperforms template and optimization baselines across production models, including a specialized defense.

  • Problem

    Prompt injection attacks require exact tool calls, but existing approaches rely heavily on hand-crafted prompts and jailbreak optimizers target generic compliance rather than parameterized actions.

    Source: §1 · p. 1

  • Method

    AutoInject uses reinforcement learning with learned comparison-based feedback to turn sparse binary attack outcomes into a dense reward for suffix optimization.

    Source: §3.2 · p. 4, §3.3 · p. 5

  • Results

    58% vs. <25% ASR on Gemini-2.5-Flash; AutoInject also reaches 21.88% ASR on Meta-SecAlign-70B, where template attacks fail outright.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AutoInject establishes an automated prompt-injection baseline and shows that learned suffixes can circumvent a model fine-tuned to resist prompt injection.

    Source: §1 · p. 2

    The attack assumes target models’ tool-calling decisions can be influenced by processed content; plan-then-execute defenses remove that opportunity but sacrifice flexibility.

    Source: §5 · p. 9

Abstract

from arXiv · show

Prompt injection is a critical vulnerability in LLM agents, yet the strongest methods still rely on human red-teamers and hand-crafted prompts. Adapting automated jailbreak optimizers does not close this gap: jailbreaks shape models toward generic compliance, while prompt injection requires emitting specific tool calls with correct parameters. The success signal is binary, and randomly sampled suffixes almost never trigger it, so standard optimizers have no gradient to follow. We present AutoInject, a black-box reinforcement learning (RL) framework that learns adversarial suffixes for prompt injection. A learned comparison-based reward scores each candidate against the best suffix seen so far, turning the binary signal into a dense reward suitable for RL optimization. The framework supports both online query-based attacks and offline-trained transferable suffixes that need no utility access at deployment, and incorporates a utility objective when task-completion feedback is available. On AgentDojo, AutoInject outperforms template attacks, GCG, TAP, and adaptive attack across production models, with statistically significant improvements under McNemar's test with p<0.05. Suffixes learned by AutoInject also break Meta-SecAlign-70B, a model fine-tuned specifically to resist prompt injection, where template attacks fail outright. The results establish an automated baseline for prompt injection and expose a gap between preference-based defenses and adaptive optimization-based attackers.

1 Introduction

Prompt injection remains difficult to automate because it requires exact tool calls, while jailbreak-style compliance objectives offer little signal for a sparse, binary attack goal. AutoInject uses comparison-based reinforcement learning to learn online and transferable suffixes, outperforming baselines while supporting utility preservation when task feedback is available.

1 Introduction

  • Jailbreak optimization targets generic compliance, but prompt injection requires a specific tool call with correctly specified parameters, making compliance a weak proxy.The injection objective is discrete over suffix tokens and has a sparse zero-one reward.Source: §1 · p. 1
  • AutoInject converts sparse binary attack feedback into a dense reward by scoring each candidate against the best suffix seen so far.Even when both candidates fail, their comparison can indicate which is closer to succeeding.Source: §1 · pp. 1–2
  • The framework supports online query-based attacks and offline-trained transferable suffixes that generalize across unseen models and tasks.The transferable suffixes need no utility access at deployment.Source: §1 · p. 2
  • 58% vs. <25% ASR on Gemini-2.5-Flash; across seven production models AutoInject beats template and optimization baselines, with improvements significant under most scenarios.AutoInject also reaches 21.88% ASR on Meta-SecAlign-70B, where template attacks fail outright.Source: §1 · p. 2
  • When task-completion feedback is available, AutoInject adds utility as a secondary reward and can preserve or improve benign-task performance under attack.The introduction reports utility under attack matching or exceeding the benign baseline.Source: §1 · p. 2

2 Related Work

Prior prompt-injection attacks rely heavily on manually crafted prompts, while automated jailbreak methods optimize a different objective. Related optimization-based attacks also use distinct access assumptions, which AutoInject’s comparisons address.

2 Related Work

  • Most existing prompt-injection techniques rely on manually crafted prompts and trial-and-error, limiting scalability and adaptability across target systems.Examples include context ignoring, fake completion, combined attacks, payload splitting, and virtualization.Source: §2 · pp. 2–3
  • Jailbreak optimizers target compliance, whereas prompt injection requires executing a parameterized action with exact argument matching, leaving a sparse binary success signal.AutoInject adapts comparison-based feedback as a dense reward for reinforcement learning.Source: §2 · p. 3
  • Prior optimization-based attacks assume different access: RL-Hammer uses an auxiliary weaker model, while Nasr et al. (2025) assumes full victim outputs.AutoInject uses black-box access and does not require token probabilities, unlike Panfilov et al. (2026).Source: §2 · p. 3

3 Method

AutoInject formulates prompt-injection suffix generation as an MDP conditioned on an injection goal and user-task context. Learned comparison feedback supplies a denser signal for GRPO despite sparse binary outcomes.

3.1 Problem Formulation

  • The attacker generates suffixes conditioned on an injection goal and user-task context, then receives terminal security feedback and optional utility feedback from the victim pipeline.Source: §3.1 · p. 3

3.2 Dense Reward via Comparison-Based Feedback

  • A learned comparison model scores each candidate against the best suffix seen so far, providing continuous preference reward even when both suffixes fail.Source: §3.2 · p. 4

3.3 Policy Optimization

  • GRPO samples completions per input and uses within-group relative advantages rather than a learned value function; clipping and KL regularization stabilize policy updates.Source: §3.3 · pp. 4–5
  • AutoInject consistently improves ASR and utility over template-based baselines across four production LLMs; utility often exceeds the benign baseline.Source: Table 1 · p. 5
  • Training runs on individual AgentDojo goal-task pairs until security and utility scores both reach 1 or the query budget is exhausted.Source: §3.3 · p. 5

4 Experiments

Experiments compare AutoInject with template, optimization, and transfer-based attacks on AgentDojo, then test the contributions of its learned policy, GRPO training, and feedback model. AutoInject performs strongly across reported settings, while cross-task transfer incurs a utility trade-off and ablations show the value of both learning components.

4.1 Experimental Setup

  • AgentDojo programmatically verifies task completion and injection success, and optimization-based baselines use query budgets equivalent to AutoInject.The benchmark covers 97 user tasks across banking, travel, workspace, and Slack.Source: §4.1 · pp. 5–6

4.2 Main Results

  • AutoInject reaches 36.57% ASR versus 13.9% for evolutionary search on GPT-5-nano; on Qwen3-4B, TAP reaches 36.6% and GCG 23.0% versus 11.2% for direct instruction.The GPT-5-nano comparison matches generator, critic, and query budget; Table 2 reports attack success rates across methods.Source: §4.2 · p. 6, Table 2 · p. 6
  • On Meta-SecAlign-70B, AutoInject achieves 21.88% ASR with utility comparable to the no-attack baseline, while template attacks are neutralized at or near 0%.The evaluation uses the hardest AgentDojo tasks.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 Transferability of Adversarial Suffixes

  • In fixed-injection transfer, AutoInject matches or exceeds RL-Hammer on four of five targets and significantly wins on GPT-4o-mini (45.0% vs. 22.9%, p < 0.05).Both methods preserve utility close to the no-attack baseline in this setting.Source: §4.3 · p. 7, Table 4 · p. 7
  • In cross-task transfer, AutoInject significantly outperforms RL-Hammer on four of five targets, but RL-Hammer wins on GPT-5-nano (25.0% vs. 10.0%).The GPT-5-nano wins are on two in-distribution banking pairs; RL-Hammer fails on the six out-of-distribution pairs.Source: §4.3 · p. 7, Table 5 · p. 8
  • Cross-task utility falls from the benign baseline for both methods, including AutoInject on GPT-4o-mini (75.0% to 37.5%), unlike fixed-injection transfer.The paper relates this trade-off to generalizing across suites and injection goals simultaneously.Source: §4.3 · p. 7

4.4 Ablation Studies

  • Without GRPO updates, ASR falls from 97.69% to 88.32% on GPT-4.1-nano and from 36.57% to 19.44% on GPT-5-nano.The relative degradation is larger on the harder target: 47% versus 10%.Source: §4.4 · pp. 8–9
  • Removing learned feedback lowers GPT-5-nano ASR from 36.57% to 25.00%; removing both feedback and GRPO lowers it to 17.36%.On GPT-4.1-nano, removing feedback changes ASR from 97.69% to 95.14%, a marginal benefit relative to the harder target.Source: §4.4 · p. 9

5 Discussions and Limitations

The discussion identifies two boundaries for AutoInject: its evaluation assumes immediate tool execution, and it targets agents whose tool decisions can be influenced by untrusted content. Deliberation and plan-then-execute designs are presented as defenses, though the latter trade away flexibility.

5 Discussions and Limitations

  • Claude Haiku 4.5’s requests for clarification break the AgentDojo pipeline, which assumes target models execute tool calls immediately.The authors suggest confirmation-seeking for sensitive operations as a possible defense orthogonal to input filtering and safety fine-tuning.Source: §5 · p. 9
  • Plan-then-execute defenses commit to an action sequence before untrusted data arrives, preventing injections from changing control flow.The authors note that this design sacrifices some of the flexibility that makes LLM agents useful.Source: §5 · p. 9

6 Conclusion

The paper concludes that RL with learned dense feedback can automate prompt injection across models while preserving utility. Its broader-impact discussion frames automated attacks as tools for defense evaluation and as capabilities that could also lower the barrier to attacking deployed agents.

6 Conclusion

  • AutoInject combines GRPO and learned dense feedback to outperform template-based and optimization baselines across frontier LLMs and specialized defenses.The conclusion reports that learned suffixes can generalize across model families while maintaining or exceeding baseline utility.Source: §6 · p. 9

A Broader Impacts

  • Automated attack generation can help defenders stress-test systems and expose gaps missed by template-only evaluation.On Meta-SecAlign-70B, AutoInject achieves 21.88% attack success where template baselines are neutralized.Source: A Broader Impacts · p. 12
  • The same capability lowers the barrier to attacking deployed agents: AutoInject needs only black-box query access and a small generator model.Transferable suffixes can be optimized offline against surrogate targets and deployed without further interaction.Source: A Broader Impacts · p. 12
  • The evaluation uses public AgentDojo rather than deployed systems, and released suffixes are tied to AgentDojo task contexts.The paper identifies confirmation-seeking behavior and plan-then-execute architectures as defensive directions.Source: A Broader Impacts · p. 12

A Experimental Details

The appendix documents policy settings, a feedback-weight schedule, the injection template, runtime, and asset access constraints. It specifies how training varies across target models and how the fixed framing and learned suffix play distinct roles.

A.1 Hyperparameter Configuration

  • Qwen2-1.5B is the attack-policy base; Meta-SecAlign uses a warmup of 4 instead of 2, while Gemma3-4B and Qwen3-4B use modified training settings.For Gemma3-4B and Qwen3-4B, training uses 32 iterations and a 28K-token context window.Source: §A.1 · p. 12

A.2 Feedback Regime Scheduling

  • The comparison-feedback weight falls from 0.8 to 0.1 as the consumed query-budget fraction moves from below 0.1 to at least 0.5.The schedule reduces learned-feedback reliance as successful attacks become more available for ground-truth evaluation.Source: §A.2 · pp. 12–13

A.3 Injection Template Structure

  • A fixed AgentDojo-style user-message template supplies social-engineering framing, while the RL policy learns the attack suffix.The template impersonates the user, invokes urgency, and promises completion of the original task afterward.Source: §A.3 · pp. 13–14

A.4 Runtime and Complexity

  • Exhausting the budget takes about 1.5 hours on one Titan RTX, with wall-clock time dominated by victim-model API latency.The matched evolutionary-search configuration takes 1.5–2 hours.Source: §A.4 · p. 14

A.5 Licenses for Existing Assets

  • Assets are used under their terms; Meta-SecAlign-70B is evaluated non-commercially under CC BY-NC 4.0, a restriction inherited by code that interacts with it.Table 9 distinguishes hosted API access from open weights.Source: §A.5 · p. 14, Table 9 · p. 14

B Evaluation Metrics

Utility under attack records whether the agent’s legitimate task remains completable in the presence of injection attempts. Its rules distinguish successful injections that preserve task completion from attacks that disrupt the task or attempts with no successful injection.

B Evaluation Metrics

  • Utility under attack is 1 if any attempt achieves both injection execution (σ = 1) and task completion (µ = 1).This captures a successful attack that leaves the legitimate task completed.Source: B Evaluation Metrics · p. 15
  • Utility under attack is 0 when an injection succeeds but no attempt achieves both injection execution and task completion.The metric treats successful attacks that necessarily disrupt the user task as utility failures.Source: B Evaluation Metrics · p. 15
  • If no attack succeeds but the agent completes the legitimate task, utility under attack is 1.This rule records that the agent remains functional when no successful attack exists.Source: B Evaluation Metrics · p. 15
  • All remaining cases receive utility under attack = 0; the metric distinguishes covertly successful attacks from attacks that disrupt the agent’s primary purpose.The paper characterizes the former as a more sophisticated and dangerous attack vector because users may remain unaware of compromise.Source: B Evaluation Metrics · p. 15

C Task Selection

The appendix specifies task selections for baseline and transfer evaluations, then compares AutoInject with a reimplemented evolutionary search under matched resources and feedback. On the full banking suite with GPT-5-nano, AutoInject achieves higher ASR, while evolutionary search remains competitive on the open-weight suite.

C Task Selection

  • Baseline comparisons use 20 task pairs per suite, while hardened-model evaluation selects the 2 hardest user tasks and 4 hardest injection tasks per suite.The baseline selection follows Hofer et al. (2025); the hardened evaluation uses 8 pairs per suite.Source: C Task Selection · p. 15, Table 10 · p. 15
  • Cross-task transfer tests 8 held-out pairs across all four suites, requiring generalization to new user tasks, injection goals, and application domains.AutoInject’s transfer strings come from five successful source pairs in the banking suite, and the same task-agnostic suffix is tested across the 8 pairs.Source: C Task Selection · pp. 15–16

D Comparison with Evolutionary Search

  • The evolutionary-search reimplementation uses MAP-Elites with five islands and a 3×3 suffix-length/diversity grid, with eight LLM-generated candidates per step.The original implementation was unavailable, so some unspecified settings were resolved using MAP-Elites defaults.Source: D Comparison with Evolutionary Search · pp. 16–17
  • The controlled comparison matches generator, critic, and query budget, and removes the original method’s victim-output access to align feedback with AutoInject’s indirect-injection setting.Both methods use Qwen2-1.5B, GPT-4o-mini, and a query budget of 260.Source: D Comparison with Evolutionary Search · p. 17
  • 13.9% ASR for evolutionary search versus 36.57% for AutoInject on the full banking suite with GPT-5-nano under matched generator, critic, and budget.On the open-weight suite, evolutionary search is competitive: 40.0% / 35.0% on Qwen3-4B / Gemma3-4B versus AutoInject’s 42.5% / 35.0%.Source: D Comparison with Evolutionary Search · p. 17

E.1 Additional Results to the Full Agentdojo Benchmark Attack

Additional AgentDojo evaluations show AutoInject outperforming template baselines on two earlier-generation models. The appendix also describes transferable “allelujah” suffixes whose success depends on token ordering and surrounding tokens.

E.1 Additional Results to the Full Agentdojo Benchmark Attack

  • 39.62% ASR on Gemini-2.0-flash, nearly 5× the best baseline’s 8.01%, while utility rises from 33.93% to 64.81%, above the benign 47.42%.On GPT-4o-mini, AutoInject reaches 50.37% ASR versus 22.87% for the strongest baseline, with utility comparable to the best template methods.Source: §E.1 · pp. 17–18, Table 14 · p. 17
  • GPT-4o-mini appears more susceptible overall than newer models, yet the learned attack policy remains effective against its defenses.The authors relate the susceptibility pattern to improved safety training in more recent models and suggest learned attacks may generalize beyond template-specific vulnerabilities.Source: §E.1 · p. 18

E.2 The “Allelujah" Attack

  • Suffix 1 attacks 70 Gemini-2.5-flash task pairs and 53 GPT-4o-mini pairs out of 949, while rearranging its tokens or changing minor phrases alters success rates.The authors hypothesize that the tokens may exploit pretraining regularities and note the suffixes might originate in their few-shot examples.Source: §E.2 · p. 18, Table 15 · p. 18

E.3 Comparison with Search-based Adaptive Attack

AutoInject is compared with adaptive random search and evolutionary search, with significance testing and confidence intervals used to assess outcomes. Its advantage over search baselines is clearest on the reported harder targets, while evolutionary search remains competitive on smaller open-weight targets.

E.3 Comparison with Search-based Adaptive Attack

  • The adaptive-search baseline’s mutation-rate sweep finds differences within about 6 percentage points, with mr = 0.3 best or tied-best across three targets.The authors use mr = 0.3 for the main comparison.Source: §E.3 · p. 18
  • 58.0% versus 34.0% ASR on Gemini-2.5-flash, and 12.6% versus 7.6% ASR with 98.5% versus 77.1% utility on Claude-Sonnet-3.5, favor AutoInject over adaptive random search.The adaptive baseline uses the same scoring but randomly mutates 30% of the best suffix’s tokens; AutoInject’s learned policy is described as generating more semantically coherent suffixes.Source: §E.3 · pp. 18–19, Table 17 · p. 19

E.4 Comparison with Evolutionary Search

  • The evolutionary-search comparison reimplements MAP-Elites and matches generator, critic, budget, and feedback access to isolate algorithmic differences.The original method assumes access to the victim’s full textual output, whereas this reimplementation uses binary success and critic scoring without that access.Source: §E.4 · p. 19
  • 13.9% ASR for evolutionary search versus 36.57% for AutoInject on the full banking suite with GPT-5-nano, although results are competitive on the open-weight suite.The comparison matches generator, critic, and query budget; the reported open-weight rates are 40.0% / 35.0% versus 42.5% / 35.0% on Qwen3-4B / Gemma3-4B.Source: §E.4 · p. 20

F Statistical Significance and Confidence Intervals

  • For AgentDojo attack-success outcomes, the appendix reports 95% Wilson score intervals for each main-text table cell and uses McNemar’s test on paired per-task outcomes to compare AutoInject with the strongest competing baseline.Source: F Statistical Significance and… · p. 20

F.1 Confidence Intervals for Table 1

The appendix reports Wilson intervals for ASR entries and McNemar tests for paired comparisons. It also describes prompts that elicit and compare candidate attack suffixes.

F.1 Confidence Intervals for Table 1

  • McNemar’s test against the strongest template baseline yields p < 0.0001 on Gemini-2.5-flash, GPT-4.1-nano, and GPT-5-nano, and p = 0.031 on Claude-Sonnet-3.5.Table 18 reports Wilson 95% intervals for Table 1 ASR entries; Table 19 reports such intervals for Table 2 with n = 80 per target.Source: §F.1 · p. 20, Table 18 · p. 20, Table 19 · p. 20

F.4 Confidence Intervals for Figure 2

  • For the adaptive-attack comparison, McNemar’s test gives p < 0.0001 on Gemini-2.5-flash and p = 0.016 on Claude-Sonnet-3.5.Table 21 supplies Wilson 95% intervals for the comparison; the appendix says the improvement is most pronounced on more robust targets.Source: §F.4 · p. 20, Table 21 · p. 21

G Prompt Templates

  • The attack-policy prompt supplies the injection goal, user-task context, few-shot suffix examples, and optionally the current best suffix; the feedback model compares two suffixes for likely attack success.The comparison model returns a binary preference, with token log-probabilities for “0” and “1” normalized by softmax.Source: G Prompt Templates · pp. 22–23
Loading 2602.05746v2…