PaperlayerView exact revision on arXiv

Source-linked AI summary

PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

Chenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia

arXiv:2603.13026v2cs.LGcs.CR

TL;DR

Prompt injection defenses remain insufficiently evaluated against adaptive attackers, especially in real-world and agentic LLM applications. PISmith trains an attack LLM with reinforcement learning and introduces mechanisms for exploration and learning from sparse successes. Across 13 benchmarks and agentic settings, it finds that state-of-the-art defenses remain vulnerable to adaptive attacks.

  • Problem

    The robustness of prompt injection defenses against adaptive attacks remains insufficiently evaluated, despite security risks to real-world LLM applications.

  • Method

    PISmith formulates red teaming as on-policy reinforcement learning that trains an attack LLM to generate injected prompts against defended target models.

  • Results

    Across 13 datasets, PISmith finds that state-of-the-art defenses remain vulnerable to adaptive attacks and also achieves strong performance in InjecAgent and AgentDojo.

  • Takeaways & Limitations

    Prompt injection defenses need greater robustness while preserving benign utility, and PISmith provides a framework for systematically stress-testing them.

  • Takeaways & Limitations

    The released red-teaming tool carries dual-use risks if misused, so the authors recommend controlled evaluation and responsible disclosure.

Abstract

from arXiv · show

Prompt injection poses serious security risks to real-world LLM applications, particularly autonomous agents. Although many defenses have been proposed, their robustness against adaptive attacks remains insufficiently evaluated, potentially creating a false sense of security. In this work, we propose PISmith, a reinforcement learning (RL)-based red-teaming framework that systematically assesses existing prompt-injection defenses by training an attack LLM to optimize injected prompts in a practical black-box setting, where the attacker can only query the defended LLM and observe its outputs. We find that directly applying standard GRPO to attack strong defenses leads to sub-optimal performance due to extreme reward sparsity -- most generated injected prompts are blocked by the defense, causing the policy's entropy to collapse before discovering effective attack strategies, while the rare successes cannot be learned effectively. In response, we introduce adaptive entropy regularization and dynamic advantage weighting to sustain exploration and amplify learning from scarce successes. Extensive evaluation on 13 benchmarks demonstrates that state-of-the-art prompt injection defenses remain vulnerable to adaptive attacks. We also compare PISmith with 7 baselines across static, search-based, and RL-based attack categories, showing that PISmith consistently achieves the highest attack success rates. Furthermore, PISmith achieves strong performance in agentic settings on InjecAgent and AgentDojo against both open-source and closed-source LLMs (e.g., GPT-4o-mini and GPT-5-nano). Our code is available at https://github.com/albert-y1n/PISmith.

1 Introduction

PISmith is introduced to systematically red-team prompt injection defenses because their robustness against adaptive attacks remains unclear. It addresses reward-sparse reinforcement-learning attacks and evaluates defenses across diverse benchmarks and agentic settings.

  • Motivation: Adaptive attacks can significantly break defenses that report near-zero attack success rates, motivating automated red-teaming tools.Existing attacks and defenses may perform well under non-adaptive evaluation while remaining vulnerable to attackers that tailor strategies to each defense.
  • PISmith: PISmith trains an attack LLM with on-policy reinforcement learning to generate effective injected prompts against defended target models.The method treats the attack as a policy optimization problem and targets strong defenses that make successful rollouts rare.
  • PISmith: Adaptive entropy regularization sustains exploration, while dynamic advantage weighting amplifies learning from rare successful rollouts.These mechanisms address entropy collapse and dilution of successful signals under extreme reward sparsity.
  • Evaluation: 13 datasets spanning question answering, retrieval-augmented generation, and long-context tasks are used to evaluate 8 defenses against 7 baselines.The evaluation is designed to compare red-teaming performance across diverse prompt-injection settings.
  • Evaluation: State-of-the-art defenses cannot simultaneously maintain high benign utility and resist adaptive attacks effectively.The reported evaluation identifies a tension between usefulness in benign settings and robustness against adaptive attackers.
  • Agentic evaluation: PISmith also demonstrates strong attack performance on InjecAgent and AgentDojo against open-source and closed-source LLMs.These agentic evaluations extend the assessment beyond the benchmark datasets.

2 Related Work

Prior work includes RL, static, search-based, and defense-oriented approaches to prompt injection security. These approaches differ in adaptability, optimization machinery, and whether they detect or prevent injected prompts.

  • Reinforcement learning: PPO established a foundation for reinforcement-learning-based LLM fine-tuning, but it requires a resource-intensive learned value model.GRPO removes this requirement by computing advantages from group-relative rewards.
  • Reinforcement learning: GRPO computes advantages from group-relative rewards and has achieved strong performance in mathematical reasoning and code generation.The paper positions GRPO as a relevant foundation for reinforcement-learning-based attack optimization.
  • Attack methods: Static attacks use predefined templates and are inherently non-adaptive, whereas search-based attacks iteratively refine injected prompts with an auxiliary LLM.These categories represent different levels of attack adaptation.
  • Defenses: Prompt-injection defenses mainly comprise filter-based methods that detect injected context and prevention-based methods that aim to stop attacks before target-model processing.Filter-based defenses employ a separate model to identify potentially injected context.

3 Problem Formulation, Challenge, and Design of PISmith

PISmith formulates prompt-injection red teaming as black-box on-policy RL for an attack LLM, then addresses standard GRPO’s reward-sparsity failures with coordinated exploration and learning mechanisms.

  • 3.1 Problem Formulation: The attacker trains an attack policy to insert prompts into task contexts and maximize binary success against a defended target LLM accessed as a black box.Success is 1 when the target response fulfills the injected task and 0 otherwise.
  • 3.2 Challenges of Standard GRPO for Prompt Injection Red Teaming: Standard GRPO is challenged because strong defenses make successful rollouts rare, while training difficulty cannot be easily adjusted through task control.The target model and deployed defense largely determine prompt-injection difficulty.
  • 3.2 Challenges of Standard GRPO for Prompt Injection Red Teaming: Sparse successes cause GRPO to overfit to isolated successful prompts, lowering policy entropy and producing increasingly homogeneous rollouts.The resulting concentration reduces opportunities to discover generalizable attack strategies.
  • 3.3 Design of PISmith: Prior multi-target training approaches can remain suboptimal because signals from undefended and strongly defended target LLMs may not match.PISmith instead develops its mechanisms around the defended-target red-teaming setting.
  • 3.3.1 Adaptive Entropy Regularization: Adaptive entropy regularization restores exploration by activating an entropy bonus only below an entropy cap and adapting its coefficient to mean reward.The coefficient is stronger when successful rollouts are scarce and decreases as reward rises above the threshold.
  • 3.3.1 Adaptive Entropy Regularization: The adaptive coefficient uses stronger regularization at low mean reward and relaxes toward a base coefficient as successful injected prompts become more common.This design targets the evolving exploration–exploitation balance during training.
  • 3.3.2 Dynamic Advantage Weighting: Dynamic advantage weighting amplifies the contribution of rare successful rollouts when mean reward is low, then decays to standard GRPO weighting as success rates improve.The mechanism is designed to prevent the majority of failed rollouts from diluting scarce success signals.
  • 3.3.3 Complete Objective: PISmith combines entropy regularization and advantage weighting so exploration produces rare successes and weighting converts them into effective policy updates.Both mechanisms are strongest when mean reward is low and relax together as training improves.

4 Evaluation

Across 13 benchmarks and agentic settings, the evaluation tests whether prompt-injection defenses resist PISmith while preserving benign utility. PISmith consistently exposes weaknesses in existing defenses, with adaptive mechanisms, broader training data, and strong transfer to complex agent tasks.

  • Evaluation setup: The evaluation spans 13 datasets covering question answering, retrieval-augmented generation, and long-context tasks, plus InjecAgent and AgentDojo agentic benchmarks.It compares defenses and attacks using ASR@10, ASR@1, and benign task utility.
  • RQ1: Adaptive attack effectiveness: Against Meta-SecAlign-8B, PISmith reaches average ASR@10 of 1.0 and ASR@1 of 0.87 across 12 unseen benchmarks after training on 100 Dolly Closed QA samples.The strongest baseline, RL-Hammer, reaches 0.70/0.48, while search-based baselines achieve average ASR@1 values of 0.11, 0.16, and 0.21.
  • RQ2: Utility–robustness trade-off: No evaluated defense simultaneously maintains high benign utility and low PISmith ASR across the 13-benchmark average.Defenses preserving utility tend to be more vulnerable, whereas aggressive defenses reduce utility by blocking malicious and benign context alike.
  • RQ3: Agentic generalization: On InjecAgent, PISmith achieves ASR of at least 0.95 across target LLMs, including 0.95 on GPT-5-nano where static attacks completely fail.On AgentDojo, it improves ASR across closed-source LLMs in multi-step workflows requiring valid tool selection, arguments, and environment-state changes.
  • Ablation studies: Removing either adaptive mechanism substantially reduces performance: vanilla GRPO reaches 0.05 average ASR@1, “w/o Entropy” reaches 0.09, and “w/o Boost” reaches 0.57.The full method reaches 0.87; entropy increases successful rollouts, while advantage weighting improves learning from sparse successes.
  • Scaling training data: Increasing training diversity raises average ASR@1 from 0.87 to 0.99 while ASR@10 remains 1.00 in the non-agent setting.The agentic study likewise investigates whether broader training coverage improves attacks against GPT-5-nano.

5 Conclusion

PISmith is a reinforcement learning-based red-teaming framework for systematically evaluating prompt injection defenses. Extensive evaluations show that state-of-the-art defenses remain vulnerable to adaptive attacks, underscoring the need for more resilient and utility-preserving defenses.

  • PISmith systematically evaluates prompt injection defenses using reinforcement learning-based red teaming.
  • State-of-the-art prompt injection defenses remain vulnerable to adaptive attacks.
  • Current LLMs and prompt injection defenses remain insufficient for secure real-world deployment.
  • PISmith may also support defense development and scaling to more capable attacker models.

Ethics Statement

The authors frame PISmith as a responsible-disclosure research tool while acknowledging dual-use and misuse risks. Experiments are confined to public benchmarks and models in a controlled environment without real-world systems, users, private data, or live harmful-content deployment.

  • PISmith is intended to evaluate prompt-injection robustness and follows responsible disclosure by targeting publicly available defenses.
  • The released attack tool carries dual-use risks if misused.
  • All experiments use publicly available benchmarks and models in a controlled research environment.
  • The evaluation involves no real-world systems, users, private data, or harmful content deployed in live applications.
  • The authors plan clear documentation and encourage controlled stress testing and appropriate reporting of findings.

A.1 Future Work: Stronger Attackers and Defensive Applications

The appendix discusses stronger attacker models, defensive applications, and why PISmith’s reward-adaptive mechanisms differ from existing reinforcement-learning approaches. It also situates PISmith among prompt-injection attacks and defenses, emphasizing the difficulty of maintaining utility against adaptive attacks.

  • Stronger Attackers: PISmith’s attack performance is expected to improve with more capable attacker models, but systematic scaling remains future work.The current experiments use Qwen3-4B-Instruct-2507, while the proposed mechanisms require no architecture-specific modification.
  • Defensive Applications: The same adaptive attacker could in principle be incorporated into defense training.
  • Reinforcement Learning: PISmith uses reward-conditioned entropy adaptation because exploration needs are high when rewards are sparse and should relax as attack success improves.
  • Reinforcement Learning: Difficulty-based sampling is unsuitable when strong defenses make nearly all initial attack rollouts unsuccessful and task difficulty cannot be readily controlled.
  • Prompt Injection Attacks: PISmith addresses reward sparsity directly without requiring training against multiple target LLMs.
  • Prompt Injection Defenses: Existing defenses cannot simultaneously maintain high task utility and resist adaptive attacks in the reported experiments.

C Details of Injected Task

The evaluation defines four injected-task categories spanning phishing, promotion, access denial, and infrastructure failure, then trains and evaluates attack methods across non-agent and agentic benchmarks. Success is judged by whether the target response follows the injected task’s main intent, with efficiency also compared by total target-model queries.

  • Injected Task Categories: The injected tasks cover phishing, content promotion, access denial, and infrastructure failure objectives.Together, these categories span motivations from financial gain to denial of service.
  • Experimental Setup: PISmith uses Qwen3-4B-Instruct-2507 as its base attack LLM and trains on benchmark-specific data splits.
  • Baselines and Training: The experiments compare PISmith with static attacks and other reinforcement-learning methods using shared attack-LLM prompt templates.
  • Evaluation: An LLM-as-judge produces a binary YES/NO decision based on whether the target response follows the injected task’s main intent.
  • Non-Agent Benchmarks: Non-agent training asks the attack LLM to embed an injected task into reference text as a legitimate update.
  • Prompt Construction: Attack outputs are required to be concise, persuasive, and wrapped in <prompt></prompt> tags.
  • Agentic Benchmarks: Agentic training asks the attack LLM to override an agent’s current task and induce a specified injection goal at a stated insertion location.
  • Efficiency: Attack-method efficiency is compared by total queries to the target model on 13 benchmarks against Meta-SecAlign-8B.

D.4 Efficiency Analysis

PISmith is evaluated by target-model query cost, the dominant expense when the defended target LLM is queried. Across training and inference, it requires fewer total queries than the compared attack methods.

  • Cost measure: Query count is used as the computational-cost measure because target-model queries dominate when the target LLM is equipped with a defense.
  • Training cost: 15,000 queries are required for PISmith training, compared with approximately 30,000 for RL-Hammer and none for search-based methods.PISmith trains on 100 cases for 10 epochs with 15 rollouts per case per epoch; RL-Hammer uses two target models.
  • Inference cost: 17,000 queries are required for each RL-based method during inference across the 1,700 evaluation cases under ASR@10.Each method generates 10 candidate injected prompts per case.
  • Total cost: 32,000 total queries make PISmith the least costly method, while RL-Hammer costs 1.5× more and search-based methods cost 2.3×–6.1× more.The search-based multipliers are TAP 2.6×, PAIR 2.3×, and Strategy 6.1× relative to PISmith.

E Full Results for Utility–Robustness Evaluation (RQ2)

The utility–robustness evaluation covers 13 benchmarks and seven defenses across prevention-based and filter-based categories. It reports utility without attack, attack success rates under several attack methods, and qualitative injected-prompt examples in non-agent and agent settings.

  • Evaluation setup: Seven defenses are evaluated across prevention-based and filter-based categories using Qwen3-4B-Instruct-2507 as the target LLM.Prevention-based defenses neutralize injected prompts without explicitly detecting them, whereas filter-based defenses use a separate model to detect and block them.
  • Defenses: The defense set includes DataSentinel, Sandwich, Instructional, PromptArmor, DataFilter, PIGuard, and PromptGuard.These methods span detection, prevention, filtering, and guardrail-based approaches described in the evaluation materials.
  • Aggregate results: Table 7 averages ASR@1 and utility across all 13 benchmarks for each defense and attack method.Utility measures task accuracy under no attack, while PISmith reports ASR@10 and ASR@1 with N = 10.
  • Per-benchmark results: The per-benchmark breakdown reports utility without attack and attack success rates for Direct, Combined, TAP, PAIR, and PISmith.Bold values indicate the best ASR per benchmark.
  • Qualitative examples: Qualitative examples show PISmith-generated injected prompts against six defenses on the same HotpotQA-Long instance and against Meta-SecAlign-8B and GPT-5-nano in InjecAgent.

G Hyperparameter Sensitivity Analysis

The default PISmith hyperparameter configuration is reported as a local optimum rather than an arbitrary choice, with both under- and over-setting harming training in interpretable ways. The same defaults are used across multiple benchmarks and defenses, while judge validation shows high agreement with human annotations.

  • Default configuration: Hcap, βmax, βbase, γmax, and τ are fixed at 0.5, 0.01, 0.001, 5.0, and 0.5 across evaluation settings.These are the default values used in the main experiments.
  • Advantage amplification: Small γmax under-weights rare successes, large γmax over-amplifies them, and both settings can reduce performance through insufficient learning or overfitting.
  • Entropy cap: Small Hcap delays entropy regularization, whereas large Hcap keeps generation overly random and harms attack coherence.
  • Reward threshold: τ = 0.5 separates failure-dominated and success-dominated rollout groups under binary rewards, after which PISmith smoothly recovers standard GRPO.
  • Cross-setting robustness: The same defaults are used across PIArena, InjecAgent, and AgentDojo without retuning, suggesting generalization across benchmarks and defenses.
  • Judge validation: Human consensus agrees with the GPT-4o-mini judge on 98% of 200 instances, with an upper-bound false-positive rate of 2%.The four disagreements were cases accepted by the judge but judged partial or ambiguous by humans.

I Comparison with General Entropy-Aware GRPO Variants

PISmith is compared with entropy-aware and sampling-based GRPO variants in the same attacker pipeline. The comparison attributes its advantage to reward-conditioned exploration that shifts from exploration under sparse rewards to exploitation after successes emerge.

  • Experimental control: Both EPO and DAPO are ported into the identical PISmith attacker pipeline for controlled comparison.
  • EPO comparison: EPO improves substantially over vanilla GRPO but still trails PISmith by a wide margin.The comparison uses the same base model, training data, evaluation protocol, and attacker pipeline.
  • Exploration schedule: PISmith shifts from maximal exploration when rewards are near zero to rapid exploitation after effective injections are discovered.This schedule differs from EPO’s fixed entropy target for sustained reasoning.
  • DAPO comparison: DAPO performs markedly worse because filtering all-failure groups under strong defenses discards most of the training batch and leaves too few useful gradient updates.PISmith retains the full batch and up-weights rare successes instead.
Loading 2603.13026v2…