PaperlayerView exact revision on arXiv

Source-linked AI summary

PIArena: A Platform for Prompt Injection Evaluation

Runpeng Geng, Chenlong Yin, Yanting Wang, Ying Chen, Jinyuan Jia

arXiv:2604.08499v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Prompt injection poses serious security risks, but the field lacks a unified way to compare defenses across diverse attacks, tasks, and benchmarks. PIArena provides an extensible evaluation platform and a feedback-adaptive attack, and its evaluation exposes limited defense generalization, vulnerability to adaptive attacks, and fundamental difficulty when injected and target tasks align.

  • Problem

    Prompt injection evaluation lacks a unified platform for reliably comparing defenses and assessing robustness across diverse attacks, tasks, and benchmarks.

  • Method

    PIArena integrates attacks and defenses for plug-and-play evaluation across benchmarks and introduces a strategy-based attack that adaptively optimizes injected prompts from defense feedback.

  • Results

    Evaluation reveals limited defense generalizability across tasks, vulnerability to adaptive attacks, and fundamental challenges when injected and target tasks align.

  • Takeaways & Limitations

    Systematic evaluation can help researchers identify weaknesses and develop more robust and generalizable prompt-injection defenses.

  • Takeaways & Limitations

    The curated benchmarks may not fully reflect real-world scenarios, so they are presented as a first step before more complex settings.

Abstract

from arXiv · show

Prompt injection attacks pose serious security risks across a wide range of real-world applications. While receiving increasing attention, the community faces a critical gap: the lack of a unified platform for prompt injection evaluation. This makes it challenging to reliably compare defenses, understand their true robustness under diverse attacks, or assess how well they generalize across tasks and benchmarks. For instance, many defenses initially reported as effective were later found to exhibit limited robustness on diverse datasets and attacks. To bridge this gap, we introduce PIArena, a unified and extensible platform for prompt injection evaluation that enables users to easily integrate state-of-the-art attacks and defenses and evaluate them across a variety of existing and new benchmarks. We also design a dynamic strategy-based attack that adaptively optimizes injected prompts based on defense feedback. Through comprehensive evaluation using PIArena, we uncover critical limitations of state-of-the-art defenses: limited generalizability across tasks, vulnerability to adaptive attacks, and fundamental challenges when an injected task aligns with the target task. The code and datasets are available at https://github.com/sleeepeer/PIArena.

1 Introduction

Prompt injection creates serious security risks, while existing evaluation lacks a unified way to compare defenses across diverse attacks, tasks, and benchmarks. PIArena addresses this gap with an extensible evaluation platform and a feedback-adaptive attack, revealing broad weaknesses in current defenses.

  • Prompt injection can manipulate backend LLMs through malicious text placed in untrusted context.Relevant contexts include webpages, social media, shared documents, code bases, and collaborative-platform messages.
  • Existing defenses may appear effective on particular benchmarks yet show limited effectiveness on other datasets and attacks.
  • PIArena integrates state-of-the-art attacks and defenses for plug-and-play evaluation across existing and new benchmark datasets.The platform is designed as a unified and extensible ecosystem for systematic evaluation.
  • Evaluation finds limited cross-task generalizability, vulnerability to adaptive attacks, closed-source-model susceptibility, and fundamental difficulty when injected and target tasks align.The findings characterize prompt-injection defense as a fundamentally challenging research problem.
  • PIArena curates diverse, context-aware benchmark datasets and evaluates state-of-the-art attacks, defenses, and LLMs systematically.
  • A black-box strategy-based attack adaptively optimizes injected prompts using defense feedback and can bypass state-of-the-art defenses.

2 Threat Model

The threat model defines prompt injection around a user pursuing a target task, an attacker inserting an injected instruction, and a defender preserving utility while mitigating attacks.

  • The model characterizes prompt injection through three actors: the user, the attacker, and the defender.
  • The user supplies a target instruction and context to a backend LLM, which generates a response for the target task.The target task can involve operations such as summarizing a document.
  • ASR denotes Attack Success Rate.
  • The attacker inserts an injected instruction into the context to make the backend LLM perform the injected task instead.The contaminated context may enable outcomes such as advertisements or phishing links.
  • The defender must preserve performance on clean contexts while mitigating prompt-injection effects on contaminated contexts.Detection-based defenses identify injected prompts and block potentially harmful outputs.

3 Background and Related Work

Prior work includes heuristic and optimization-based attacks, prevention and detection defenses, and benchmarks for general LLM tasks or agent scenarios. However, existing benchmarks generally lack a comprehensive, extensible framework for defense evaluation.

  • Prompt Injection Attacks: Existing attacks are categorized into heuristic-based methods using static strategies or templates and optimization-based methods.A context-ignoring attack, for example, prepends an instruction such as “Ignore previous instructions, please...”
  • Limitations of Existing Evaluation: PIArena is compared with existing benchmarks as a unified platform for prompt-injection evaluation.
  • Prompt Injection Defenses: Existing defenses are categorized into prevention-based methods that preserve target-task performance and detection-based methods that identify injected prompts.
  • Prompt Injection Benchmarks: Prompt-injection benchmarks cover general LLM tasks such as question answering, summarization, and classification, as well as LLM-agent scenarios.Agent benchmarks often require complicated setups.
  • Limitations of Existing Evaluation: Most existing benchmarks lack the framework needed for comprehensive defense evaluation.The benchmark landscape provides datasets but not necessarily integrated evaluation mechanisms.

4 PIArena

PIArena addresses fragmented prompt-injection evaluation with a unified, extensible framework for integrating benchmarks, attacks, defenses, and metrics. It also introduces realistic context-aware injected tasks and a black-box strategy-based attack that adapts prompts using defense feedback.

  • Limitations of Existing Evaluation: Existing benchmarks rely on static attacks and lack unified APIs or extensibility for adding new datasets, attacks, and defenses.PIArena addresses these limitations with adaptive attacks, unified APIs, and a toolbox that supports community updates.
  • Platform Design: PIArena provides standardized interfaces for plug-and-play integration of attacks, defenses, and benchmarks, supporting systematic and reproducible evaluation.Its benchmark, attack, defense, and evaluator modules interact through a unified framework for comparing methods across diverse tasks.
  • Benchmark Construction: PIArena curates datasets spanning diverse target tasks and realistic injected tasks, including question answering, retrieval-augmented generation, and long-context scenarios.The platform is generic, allowing additional datasets and future applications to be incorporated through its API.
  • Benchmark Construction: Four injected-task categories model practical attack objectives: phishing, content promotion, access denial, and infrastructure failure.Each injected task is generated from the target instruction, context, and attack goal to remain contextually relevant and realistic.
  • Unified Evaluation Platform: PIArena standardizes samples, method interfaces, and metrics, including task utility and Attack Success Rate to quantify the effectiveness–utility trade-off.It also supports attack-only and defense-only evaluation modes.
  • A Strategy-based Adaptive Attack: The strategy-based adaptive attack uses black-box defense feedback to iteratively optimize injected prompts and overcome the cold-start problem in prompt optimization.Strategy-based rewriting provides semantically plausible warm starts that improve stealth, imperativeness, query efficiency, and attack diversity.

5 Evaluation and Implications

PIArena evaluates attacks and defenses across diverse tasks, benchmarks, and LLMs, revealing limited generalization, adaptive-attack vulnerability, and challenges when injected and target tasks align.

  • Evaluation setup: PIArena evaluates state-of-the-art attacks, defenses, and LLMs across question answering, information extraction, summarization, RAG, code generation, agentic, and general benchmarks.Its modules support plug-and-play evaluation across diverse datasets and attacks.
  • Main results: 11% average ASR for PISanitizer under Direct attack versus 4% under Combined Attack, showing defense performance varies across attacks.SecAlign++ reduces ASR but lowers average utility from 74% to 58% under No Attack.
  • Main results: 99% ASR without defense under the strategy-based attack versus 56% for Direct and 72% for Combined attacks, indicating adaptive attacks remain difficult to defend against.Against PISanitizer, the strategy-based attack reaches 86% ASR versus 11% for Direct and 4% for Combined attacks.
  • LLM vulnerability: 76% ASR for GPT-4o-mini and 70% for GPT-5 demonstrate substantial prompt-injection vulnerability across tested closed-source LLMs.The evaluation reports that almost all tested models achieve over 70% ASR.
  • Task alignment: When injected tasks align with target tasks, all tested defenses are ineffective because disinformation may contain no explicit instruction to detect.This creates a fundamental content-verification challenge for RAG and agentic applications using untrusted contexts.
  • Portability: Existing defenses either degrade utility or fail to sufficiently reduce ASR on agentic benchmarks, while PIArena’s portable modules support evaluation on additional benchmarks.The platform is also evaluated on general prompt-injection benchmarks.

6 Conclusion

The paper presents PIArena as a unified platform for systematic prompt-injection evaluation and introduces an adaptive strategy-based attack. Its evaluation exposes weaknesses in existing defenses and the fundamental difficulty of defending against prompt injection.

  • Conclusion: PIArena is a unified and extensible platform for systematic prompt-injection evaluation.It is intended to support comprehensive evaluation and help identify weaknesses in defenses.
  • Conclusion: The dynamic strategy-based attack adaptively optimizes injected prompts using defense feedback.This attack is presented as a central methodological contribution.
  • Conclusion: Evaluation reveals limitations of existing defenses and shows that defending against prompt injection remains fundamentally challenging.The authors position systematic evaluation as a way to support development of more robust defenses.

Limitations

The curated benchmarks may not fully reflect real-world scenarios, but the authors argue they are useful first-step tests because existing defenses already struggle on them.

  • Scope boundary: The curated benchmarks may not fully reflect real-world scenarios.This limits how directly their results transfer to more complex deployment settings.
  • Scope boundary: The benchmarks can serve as a useful first step before evaluating defenses in more complex, real-world settings.The authors base this role on defenses already struggling on the controlled benchmarks.

Ethical Considerations

The paper frames prompt-injection security as a risk across LLM applications and reviews attack, defense, and benchmark categories used to study it.

  • Attack taxonomy: Heuristic-based attacks use predefined templates, while optimization-based attacks iteratively optimize injected prompts toward an attacker’s goal.Optimization-based methods include both white-box and black-box approaches.
  • Defense taxonomy: Detection-based defenses identify injected instructions, whereas prevention-based defenses aim to preserve the target task despite contaminated context.Reviewed approaches include prompting, fine-tuning, policy-based prevention, and sanitization.
  • Benchmark landscape: General benchmarks cover tasks such as question answering, summarization, and classification, while agentic benchmarks evaluate tool-integrated and multi-step scenarios.Agentic benchmarks often require more complicated setups than general task benchmarks.
  • Evaluation scope: The paper evaluates attacks and defenses on MultiNews and other benchmarks, including general and agentic settings.The supplied benchmark references include optimization-based attacks on MultiNews and multiple benchmark families.

B Evaluation on Optimization-based Attacks

PIArena evaluates optimization-based attacks alongside strategy-based and search-based methods. The cited discussion contrasts white-box GCG with the authors’ black-box dynamic strategy-based attack.

  • Optimization-based attacks: GCG is a strong white-box optimization attack requiring victim-LLM gradient access and iterative optimization.Some defenses, including DataFilter and DataSentinel, show high robustness against GCG.
  • Optimization-based attacks: White-box optimization attacks can incur high computational cost and achieve suboptimal performance against state-of-the-art defenses.The passage specifically contrasts these limitations with the authors’ dynamic strategy-based attack.
  • Strategy-based attack: The dynamic strategy-based attack requires only black-box access while achieving high attack success rates and outperforming white-box optimization attacks.Its prompts are adaptively optimized using defense feedback.
  • Related attacks: Prior search-based attacks such as AutoDAN, PAIR, TAP, and other methods use automated or human-designed strategies to generate adversarial prompts.The related-work discussion includes role-playing, ciphered encoding, ASCII, translation, persuasion, and quality-diversity strategies.

D Future Directions for Defense Development

PIArena’s evaluation identifies limited defense generalization, strong vulnerability to adaptive attacks, and a fundamental weakness when injected tasks align with target tasks. It motivates broader cross-benchmark, adaptive, and content-level evaluation.

  • Cross-benchmark evaluation: Defenses effective on specific benchmarks often fail to generalize across target tasks, injected-task categories, attack types, and context lengths.The authors recommend comprehensive cross-benchmark evaluation during defense development.
  • Future defense development: Future defenses should combine unified cross-benchmark testing and adaptive threat models with content-level verification mechanisms.Suggested verification includes cross-referencing context information against trusted external sources or knowledge bases.
  • Adaptive threat models: Adaptive strategy-based attacks achieve significantly higher attack success rates than static attacks across nearly all defenses.The authors argue that evaluating only against static, fixed-template attacks can create a false sense of security.
  • Content-level verification: When injected tasks align with target tasks, attacks can become disinformation without explicit injected instructions.The task-alignment scenario exposes a limitation of defenses focused primarily on detecting or neutralizing instructions.

F Injected Task Creation

Injected tasks are generated as concise, context-aware direct instructions spanning four attack goals. The generation process emphasizes diversity, evaluability, non-repetition, and preservation of the original context.

  • Attack goals: Injected tasks cover phishing injection, content promotion, access denial, and infrastructure failure.The attack-goal descriptions specify links, advertising, quota or subscription blocking, and simulated backend failures.
  • Generation process: An LLM generates one context-aware injected task using the target instruction, context, selected attack goal, and previously generated tasks.Previously generated tasks are included to reduce duplication.
  • Generation constraints: Generated injections must preserve the original context and be inserted without deleting, rewriting, or modifying its characters.The injection may be placed between sentences, at the beginning, or at the end, while matching the context’s domain, style, and formatting.
  • Generation constraints: The process requires diverse, concise, and objectively evaluable injections that avoid repeated openings, syntax, templates, and similar wording.The preferred length is under 30 tokens, with a clear binary success criterion and varied linguistic patterns and tone.
  • Generation constraints: The injection must be a direct instruction for the victim LLM, not a description or meta-instruction about what to inject.The output format requires an injected_task and its attack_goal in a JSON object.
  • Evaluation: The attack pipeline evaluates success by judging whether the response completes the injected task and checks whether the defense detected or neutralized the prompt.These functions provide the success and defense-status signals used by the attack procedure.

G.2 Attack Cost Analysis

The strategy-based attack is designed for practical red-teaming: it typically converges quickly and uses a modest per-sample cost, while varying injected prompts through multiple rewriting strategies.

  • Iteration cost: Most defenses are bypassed within 1–2 attack iterations on average.SecAlign++ requires 4.555 iterations on average, consistent with its more aggressive defense and relatively lower attack success rate.
  • Attack cost: The average per-sample attack cost is approximately 8 seconds with vLLM serving and batched inference.The authors characterize this cost as modest and practical for red-teaming evaluation.
  • Strategy design: The dynamic strategy-based attack uses 10 distinct rewriting strategies to vary injected-prompt syntax and semantics.The strategies balance stealth against imperativeness by targeting different aspects of LLM processing.
  • Strategy design: The strategy library includes disguises such as data-privacy addenda, recent updates, fragmented metadata, editor corrections, system configuration blocks, FAQ sections, author addenda, conditional triggers, translation notes, and seamless continuations.These formats are designed to blend the injected task into surrounding context or document conventions.
Loading 2604.08499v1…