Defending Against Prompt Injection With a Few DefensiveTokens

1 Introduction

• The proposal addresses the gap that flexible test-time defenses are less effective than training-time defenses, despite developers needing to balance security and utility.Source: §1 · p. 1
• DefensiveToken inserts security-optimized special-token embeddings without changing model parameters, so developers can choose whether to apply the defense at test time.The provider releases the tokens alongside the LLM.Source: §1 · pp. 1–2
• Using a few tokens provides prompt-injection robustness with minimal utility loss, while omitting them leaves the LLM system operating as without the defense.The reported utility loss is smaller than for all baselines and is confined to developers who choose security.Source: §1 · pp. 1–2
• 0.24% ASR on the >31K-sample benchmark matched training-time defenses (0.20%–0.51%) and beat three test-time alternatives (>11.0%), averaged across four models.The benchmark evaluated manually designed prompt injections.Source: §1 · p. 2
• Against stronger optimization-based prompt injection, DefensiveToken lowered average ASR from 95.2% to 48.8%; the strongest test-time baseline had around 70% ASR and significant utility loss [53].Source: §1 · p. 2
• On an agentic tool-calling benchmark, DefensiveToken reduced average ASR by 5 times, compared with 2 times for the best evaluated test-time baseline.Source: §1 · p. 2

2 Related Work

• Prompt-injection attacks include heuristic optimization-free attacks and stronger optimization-based attacks that generally require white-box access and computationally heavy optimization.Source: §2 · p. 2
• Defenses include test-time detection, prompting, and system-level approaches, while training-time methods optimize models for stronger injection resistance.Detection-based defenses reject potentially malicious queries; system-level defenses target some control-flow attacks.Source: §2 · p. 2
• Training-time approaches include fine-tuning methods such as StruQ, SecAlign, and ISE, while instruction hierarchy specifies that higher-priority instructions should be obeyed.Jatmo fine-tunes on one task without task instructions, so the defended model lacks instruction-following ability.Source: §2 · p. 2
• Prompt tuning inserts trainable continuous vectors and requires white-box gradients, whereas hard prompt optimization refines discrete prompts and generally uses black-box interaction.Source: §2 · p. 2
• Jailbreak mitigation targets a malicious user, unlike this paper’s prompt-injection setting, where the user and system are benign and the environment is malicious.Source: §2 · p. 3

3.1 Preliminaries

• An LLM-integrated input separates a trusted user instruction from untrusted external data using delimiters such as [INST], [DATA], and [RESP].An attacker can insert new instructions into the external data.Source: §3.1 · p. 3
• The threat model allows attackers to know the benign instruction, prompt format, and DefensiveToken embeddings, but not to modify the tokens.The security objective is to ignore injected instructions while preserving useful responses whether or not an injection is present.Source: §3.1 · p. 3

3.2 Motivation

• Existing defenses do not jointly offer flexibility, security, and utility: training-time defenses cannot be stripped off, prompting is limited, and detectors or system-level defenses can sacrifice utility.The paper identifies LoRA as a close alternative but says token-based deployment needs no provider infrastructure changes.Source: §3.2 · p. 3
• DefensiveToken uses optional security-optimized special tokens: developers prepend them when security is needed and omit them to retain the undefended LLM’s performance.The provider optimizes and releases the tokens alongside the model for developers to apply according to their needs.Source: §3.2 · p. 3

3.3 Methodology

• The provider optimizes token embeddings while keeping model parameters unchanged, prepending the embeddings to the original input when the defense is used.The method applies gradient descent using the StruQ loss.Source: §3.3 · p. 3
• Training keeps half the samples unchanged and injects two variants into the other half with equal probability, using undefended-model responses as labels to preserve output style and utility.Source: §3.3 · p. 3

3.4 Connection to Prompt Tuning

• DefensiveToken extends prompt tuning from a shared task instruction to security across different instructions while retaining utility.The reported optimization involves approximately 20k floating-point variables and does not change LLM parameters.Source: §3.4 · p. 4

4.1 Setup

• Experiments optimize five tokens for one epoch on four open-weight models and evaluate utility and attack success across prompt-injection benchmarks.Utility is assessed with AlpacaEval2; security uses ASR, generally judged by an LLM on held-out instructions and injections.Source: §4.1 · pp. 4–5

4.3 DefensiveToken vs. Training-Time Defenses

• TaskTracker average ASR is 0.24% for DefensiveToken, close to 0.20%–0.51% for training-time defenses on optimization-free attacks.DefensiveToken is slightly weaker than training-time alternatives on optimization-based attacks and InjecAgent.Source: Figure 2 · p. 6, §4.3 · p. 7

4.2 DefensiveToken: A SOTA Test-Time Defense

• Five tokens cut optimization-free ASRs by an order of magnitude on AlpacaFarm, SEP, and TaskTracker, threefold on CyberSecEval2, and fivefold on InjecAgent.Against adaptive GCG, average ASR falls about twofold while existing test-time alternatives nearly fail.Source: §4.2 · p. 6

4.5 Ablation Study

• Falcon3-7B-Instruct ASR falls from 70% (1 token) to 0% (20 tokens), but 20 tokens cost 2.4% utility; five tokens suffice for all four models.Llama3-8B-Instruct and Llama3.1-8B-Instruct gain no benefit from tuning more than one embedding.Source: Table 4 · p. 7, §4.5 · p. 7
• Random initialization outperforms text-embedding heuristics, which the authors attribute to larger initial embedding magnitudes that ease optimization.They contrast this finding with prior prompt tuning, where text-embedding initialization worked best.Source: Table 5 · p. 7, §4.5 · p. 7

4.4 Analyzing the Utility-Security Trade-Off

• DefensiveToken loses the least utility among evaluated baselines and is closest to an ideal defense on optimization-free attacks across AlpacaFarm and SEP.Its utility loss applies only when developers choose to use the tokens.Source: §4.4 · p. 7, Figure 3 · p. 8

4.5 Ablation Study

• SecAlign loss achieves perfect security but significantly hurts utility; placing DefensiveTokens at the input start performs better than placing them at the end.The authors use StruQ loss in their design.Source: Table 6 · p. 9, §4.5 · p. 9, Table 7 · p. 9
• Learning rate 0.1 is a good security-utility choice, while 0.01 provides inadequate security and 1 can destabilize training unpredictably.Learning rate strongly affects security but not utility.Source: Table 8 · p. 9, §4.5 · p. 9

• Compared with Reminder or Sandwich, DefensiveToken reduces attack success rate by two times to an order of magnitude and has comparable robustness to StruQ and SecAlign.Source: §5 · p. 9
• DefensiveToken lets developers prioritize security or utility by choosing whether to apply the defense.Source: §5 · p. 9
• Its protection is limited to benign user instructions paired with malicious application-retrieved data, not user-malicious settings such as jailbreaks.Source: §5 · p. 9
• Although AlpacaEval2 shows a trivial utility drop, utility on more labeled datasets remains unknown, supporting the option to omit DefensiveTokens when utility is prioritized.Source: §5 · p. 9