InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models

• Prompt injection attacks exploit natural-language inputs to hijack LLM goals or leak sensitive data.Source: §1 · p. 1
• Prompt guards offer lightweight, efficient screening, but trigger-word shortcuts can make them flag harmless requests and disrupt access in interactive or medical applications.Source: §1 · pp. 1–2
• 339 benign trigger-word samples make up NotInject, and existing prompt guards score below 60% over-defense accuracy on it, near the 50% random-guessing level.Source: §1 · p. 2
• Over 83% average accuracy and a 30.8% lead over the open-source runner-up are reported for InjecGuard across benign, malicious, and over-defense inputs.The lightweight DeBERTa-based model is reported to perform similarly to GPT-4o.Source: §1 · p. 2

• Prior prompt injection research studies attack methods, algorithms, benchmarks, and datasets, alongside public discussion of risks to commercial LLMs.Source: §2 · p. 2
• Prompt guards screen inputs efficiently without another victim-LLM inference, but many models have closed training data or undisclosed implementation details.Source: §2 · p. 3
• Keyword-triggered shortcuts lead several prompt guards to classify benign inputs as malicious; this paper pairs systematic evaluation with an open-source training approach.Source: §2 · p. 3
• Injection detection combines semantic and pattern cues, making it more vulnerable to spurious correlations than semantic-only tasks and difficult to debias without losing performance.Source: §2 · p. 3
• The attention visualization contrasts ProtectAIv2’s concentration on “ignore” with InjecGuard’s attention across the sentence and benign prediction.Source: Figure 3 · p. 3

3.1 The Over-defense Issue

• Over-defense is the tendency to predict malicious labels when benign sentences contain trigger words commonly used in prompt injection; ProtectAIv2 and PromptGuard misclassify an example containing “ignore.”Source: §3.1 · p. 3

3.2 Construction of NotInject

• NotInject is built through trigger-word identification, refinement, and corpus generation to evaluate over-defense systematically.Source: §3.2 · pp. 3–4
• Candidate triggers are found by comparing word-frequency ranks in malicious and benign corpora, then filtered with an LLM and expert review.Source: §3.2 · p. 4
• The generated benchmark has 339 reviewed benign sentences, split evenly across one, two, and three trigger words and spanning varied topics and languages.Source: §3.2 · pp. 4–5

3.3 Evaluations on NotInject

• None of the evaluated guards achieves high accuracy on malicious, benign, and over-defense inputs simultaneously; open-source guards score below 60% on over-defense, while LakeraGuard and LlamaGuard3 have suboptimal malicious accuracy.Source: Figure 5 · p. 5, §3.3 · p. 5

4 InjecGuard

• InjecGuard is designed to improve malicious, benign, and over-defense accuracy, followed by a data-collection and augmentation process and a new training strategy.Source: §4 · p. 5

4.1 Data Collection and Augmentation

• The training corpus combines 20 open-source datasets and added examples in 17 underrepresented formats, yielding 61,089 benign and 15,666 injection samples.435 samples were generated through data-centric augmentation.Source: §4.1 · p. 6

4.2 Mitigating Over-defense for Free (MOF)

• MOF identifies biased tokens by testing vocabulary items individually after initial training, without relying on a dedicated over-defense dataset.Source: §4.2 · p. 6
• MOF uses the identified tokens to generate and refine 1,000 benign examples, adds them to training data, and retrains InjecGuard from scratch.Source: §4.2 · p. 6
• The retrained model focuses on the whole input’s meaning rather than trigger words and achieves high accuracy across all three dimensions.Source: §4.2 · p. 6

5.1 Experimental Setups

• Evaluation uses PINT and WildGuard benign data, PINT and BIPIA attack data, NotInject for over-defense, accuracy, and inference-cost measures.InjecGuard uses DeBERTaV3-base; baselines include prompt guards and prompted LLMs.Source: §5.1 · pp. 6–7

5.2 Main Results

• 83.48% average accuracy lets InjecGuard outperform LakeraGuard by 6.25% and ProtectAIv2 by 30.83%.Table 1 compares model performance and overhead.Source: Table 1 · p. 7, §5.2 · p. 7
• Fmops, Deepset, and PromptGuard reach only 5.60%, 5.31%, and 0.88% over-defense accuracy; InjecGuard exceeds ProtectAIv2 by 54.17%.ProtectAIv2's over-defense accuracy is 56.64%, close to random guessing at 50.00%.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 Ablation Studies

• Data-centric augmentation raises benign accuracy from 78.53% to 81.36% and malicious accuracy from 70.17% to 75.95%, but lowers over-defense accuracy from 75.22% to 64.31%.The ablation compares training designs across average, over-defense, benign, and malicious accuracy.Source: §5.3 · p. 8, Table 2 · p. 8
• MOF with retraining from scratch yields 81.89% average accuracy on the basic dataset; adding augmentation achieves the highest average accuracy, 83.48%.With both components, over-defense, benign, and malicious accuracies are 87.32%, 85.74%, and 77.39%, respectively.Source: §5.3 · p. 8

5.4 Comparison with Shortcut Mitigation

• The shortcut-mitigation baseline reduces malicious performance by 10.42%, while MOF achieves 87.32% overall defense performance and 77.39% malicious performance.The comparison is reported in Table 3.Source: §5.4 · p. 8, Table 3 · p. 8

• The paper introduces NotInject to evaluate over-defense in prompt guard models.The conclusion also presents InjecGuard as a model that outperforms existing models in performance and robustness.Source: §6 · p. 9
• The authors describe InjecGuard as the first fully open-source prompt guard against injection, including training data, strategies, code, and model.Source: §6 · p. 9
• The authors expect the release to promote transparency and foster open-source academic research on LLM safety.Source: §6 · p. 9

Limitations

• The work reports significant improvement in mitigating over-defense in prompt guard models.Source: Limitations · p. 9
• NotInject may underrepresent domain-specific benign inputs, potentially underestimating over-defense in sensitive fields such as healthcare or finance.The authors plan to add domain-specific data through industry collaboration.Source: Limitations · p. 9
• The authors say NotInject’s current design is sufficient to reveal over-defense in existing prompt guard models.They frame this as evidence of a need for improved approaches.Source: Limitations · p. 9

Ethics Statement

• The authors state that all data are synthetic or publicly sourced and involve no personal or sensitive information.They say this safeguards privacy and complies with ethical data-use standards.Source: Ethics Statement · p. 9
• The authors acknowledge that defensive security research may have dual-use potential.Source: Ethics Statement · p. 9
• They encourage using NotInject to improve LLM security rather than for malicious purposes.Source: Ethics Statement · p. 9
• The authors say open-sourcing the dataset and model supports transparency, replication, verification, and further work.Source: Ethics Statement · p. 9

A.1 Benign dataset

• The benign data combine an augmented over-defense dataset with data from 14 open-source datasets.Sources include instruction, conversation, safety, and prompt-injection datasets.Source: §A.1 · p. 14

A.2 Malicious dataset

• The malicious data draw on 12 open-source datasets and an augmented dataset.Source: §A.2 · p. 14
• The malicious augmented dataset includes long-tail formats such as XML, HTML, and Markdown.The paper points to a table for the distribution of data types.Source: §A.2 · p. 14

B.1 Full results

• The full-results section describes evaluations across NotInject, Wildguard, PINT-benchmark, and BIPIA, with results reported in Tables 1 and 7.Table 1 covers over-defense, benign, and malicious dimensions; the supplied passage gives no values for the comparisons.Source: §B.1 · p. 14

B.2 Ablation study of MOF sampling scale

• 83.48% average accuracy is reported as the best balance when MOF generates 1,000 benign samples.The authors select 1,000 as the sampling scale after observing a trade-off between over-defense and malicious accuracy as sample count increases.Source: §B.2 · p. 14

B.3 The effect of MOF on general over-defense scenarios

• 91.15% accuracy on the PINT hard-negative dataset is reported as evidence of robustness across general over-defense scenarios.The dataset serves as a substitute for evaluation in non-trigger-word-based over-defense scenarios, for which the authors say no benchmark exists.Source: §B.3 · pp. 14–15

C.2 The robustness of MOF when incorrectly identified words

• Misidentified trigger words can form benign training sentences without noisy labels or harm to basic performance, though over-defense gains may be slightly smaller.Source: §C.2 · p. 15

D.1 Word-based Generation Prompt

• LLMs generate benign sentences from trigger words or tokens for both NotInject and MOF training.Source: §D.1 · p. 15, Figure 11 · p. 17

D.2 Refinement Prompt

• GPT-4o and Llama-2-chat use the same prompts to detect malicious content in trigger-based generated sentences and check harmlessness.Source: §D.2 · p. 15, Figure 12 · p. 17

D.3 Long-tail Format Augmentation Prompt

• LLM-based augmentation targets long-tail formats in the dataset, including CSV and XML.Source: §D.3 · pp. 15–16, Figure 13 · p. 17

C.1 Visualization of NotInject dataset

• PromptGuard and ProtectAIv2 classify a safe input as an injection with high confidence, while InjecGuard classifies it as safe.The comparison is presented as a visualization of model predictions.Source: §C.1 · p. 15, Figure 7 · p. 16