Defenses Against Prompt Attacks Learn Surface Heuristics

1 Introduction

• Supervised fine-tuning defenses train on benign-versus-attack labels and commonly use explicit injection patterns such as instruction overrides and delimiter escapes.These defenses can achieve high attack rejection rates on standard benchmarks.Source: §1 · pp. 1–2
• Fine-tuned defenses rely on narrow structural regularities rather than harmful intent, rejecting benign content based on later position, attack-associated tokens, or unfamiliar topics.The paper formalizes these behaviors as position bias, token trigger bias, and topic generalization bias.Source: §1 · p. 2
• Controlled diagnostic settings keep semantic intent benign while varying only position, token identity, or topical domain.This isolates the contribution of each factor to the observed rejection or performance changes.Source: §1 · p. 2
• After defense fine-tuning, suffix-task rejection rises from below 10% to as high as 90%; one trigger token raises false refusals by up to 50% versus matched non-trigger controls, and defended models lose up to 40% test-time accuracy across diverse reasoning benchmarks.Source: §1 · p. 2
• The study characterizes three shortcut failure behaviors and evaluates them across two base models and multiple defense pipelines.The authors report that conventional benchmarks can hide safety–utility trade-offs.Source: §1 · p. 2

2.1 Prompt Injection

• Prompt injection includes direct attacks through user-visible inputs and indirect attacks embedded in retrieved content that influence downstream model behavior.Both forms involve adversarial instructions competing with intended instructions.Source: §2.1 · p. 2

2.2 Model-level Prompt Injection Defenses

• Model-level defenses include generation-time safety reasoning, training-data or objective changes, structured prompting, fine-tuning, and instruction-hierarchy methods.The cited examples include Constitutional AI, deliberative alignment, StruQ, SecAlign, Meta SecAlign, and instruction-hierarchy approaches.Source: §2.2 · pp. 2–3

3.1 Problem Statement

• Fine-tuning on narrow attack patterns may let a model minimize its objective by relying on surface features instead of malicious intent, causing rejection of entirely benign prompts.The defense dataset assigns benign or attack labels to prompts, while attack examples may share structural patterns such as position, strings, or topics.Source: §3.1 · p. 3

3.2 Failure Hypothesis

• The failure hypotheses predict that benign suffix tasks, prompts containing attack-associated strings, and prompts from unseen topics can respectively face increased rejection or degraded performance.For position bias, the suffix task is benign even though its rejection probability may be high.Source: §3.2 · p. 3
• Defended models produce shorter outputs on benign reasoning tasks, a pattern the paper links to truncated reasoning and utility loss.The figure compares generated-token-length histograms for Llama-3 Base and StrucQ Defended Llama-3 across GPQA and MMLU.Source: Figure 2 · p. 4

3.3 Diagnostic Dataset Design

• The diagnostic datasets isolate position, token identity, and topical familiarity while keeping prompts benign; the position test finds frequent rejection of the benign second task despite low rejection on single tasks.The keyword set compares a trigger-token insertion with a matched random technical term, while the topic set compares matched tasks from training and held-out domains.Source: §3.3 · p. 4

4.1 Experimental Setup

• The setup compares instruction-tuned Llama 3 and Mistral models and evaluates fine-tuning defenses alongside external prompt guards.The fine-tuning defenses include SecAlign and StrucQ; the prompt-guard set includes ProtectAIv2, LakeraGuard, PromptGuard2, Deepset, and FMOPS.Source: §4.1 · pp. 4–5

4.2.1 Position Stress Set (H1)

• Position-stress tests use GPQA, MMLU, and AIME questions presented alone or as concatenated benign tasks, with a five-question Mistral–StrucQ ablation.The five-question ablation examines behavior by question position.Source: §4.2.1 · p. 5, Figure 3 · p. 6

4.2.2 Keyword Trigger Set (H2)

• The trigger study compares standard benign prompts with benign prompts augmented by attack-associated tokens, evaluating models and guards in a unified setting.The baseline combines benign PINT and WildGuard data; the trigger set uses benign prompts with three attack-associated tokens.Source: §4.2.2 · p. 6

4.2.3 Topic Generalization Set (H3)

• The generalization test measures accuracy on benign GPQA, MMLU, and AIME tasks outside the narrow meta-instructional topics represented in defense training data.It introduces no attack-like patterns and measures utility rather than refusal rates.Source: §4.2.3 · pp. 6–7

4.3 Evaluation Metrics

• Refusal rate records refusal-style model responses or unsafe labels from guards, while shortcut metrics compare positional refusal, trigger-induced refusal gaps, and benchmark accuracy drops.Utility is task accuracy on benign benchmarks.Source: §4.3 · p. 7

4.4 Main Results

• Suffix-task rejection reaches 90%, trigger insertion raises refusal rates by 40–50%, and defended models show accuracy drops across reasoning benchmarks.The position and trigger effects occur on benign inputs; topic-generalization degradation is also reported despite the absence of adversarial instructions.Source: Table 2 · p. 5, Table 3 · p. 6, Table 4 · p. 6

4.5 In-Depth Analysis

• SFT analysis links shortcut learning to binary attack-versus-safe labels and recurring surface cues, while trigger-removal ablations provide causal evidence for token-driven rejection by prompt guards.Defense-tuned models also produce shorter benign-task outputs, and concatenated benign questions show increasingly frequent rejection of later tasks.Source: §4.5 · p. 8

4.6 Qualitative Study

• The qualitative study compares original and defended models on identical benign prompts to illustrate shortcut behavior in individual responses.Source: §4.6 · p. 8

• Counterfactual benign examples that resemble attacks may reduce lexical or positional cue reliance, but their effect on the attack-rejection–benign-utility trade-off remains unclear.Source: §5 · p. 8
• Interactive or multi-turn tests and measures such as reasoning depth or calibration could extend evaluation beyond static benchmarks and accuracy.Source: §5 · p. 8

• Controlled diagnostics identify position, token-trigger, and topic-over-rejection biases that associate defense behavior with surface cues rather than malicious intent.Source: §6 · p. 8
• The reported shortcut effects also appear in external prompt guards trained on attack-heavy data.Source: §6 · p. 8
• Aggregate rejection metrics are insufficient to characterize defense behavior on benign use cases, motivating intent-aware evaluations that preserve utility.Source: §6 · p. 9

• The findings are bounded by the selected models, benchmarks, defenses, and interpretable measures of refusal and task performance; alternative designs or deployment contexts may differ.The authors frame the analysis as complementary to, rather than exhaustive of, work on robust and intent-aware defenses.Source: §7 · p. 9

8 Ethical considerations

• The study uses benign prompts and public benchmarks, reports aggregate behavioral results, and proposes no attacks or harmful-content generation.Its stated aim is to improve visibility into safety mechanisms and inform evaluation and design choices for balancing protection with legitimate access.Source: §8 · p. 9

A.1 Prompt Injection Attacks

• Direct injection supplies adversarial instructions through user-visible input, while indirect injection embeds them in external content retrieved by a benign system.Both can introduce text that competes with or overrides system instructions.Source: §A.1 · p. 11

A.2 Model-level Defenses

• Model-level defenses include safety-principle training, inference-time policy reasoning, structured prompt formats, preference optimization, and instruction-hierarchy training.Examples include Constitutional AI, deliberative alignment, StruQ, SecAlign, Meta SecAlign, and instruction-hierarchy methods.Source: §A.2 · p. 11
• The paper analyzes how supervised defense tuning can produce shortcut behaviors associated with false refusals, suppressed reasoning, and degraded generalization on benign inputs.This extends prior observations of trade-offs between security and utility.Source: §A.2 · p. 11

B.1 Experimental Setup

• The experiments use instruction-tuned Llama 3 and Mistral models without additional task-specific fine-tuning to assess behavior under defenses and trigger conditions.The paper characterizes Llama 3 as strong in instruction following and stable formatting, and Mistral as lightweight with competitive safety and reasoning performance.Source: §B.1 · p. 11
• SecAlign and StrucQ modify model parameters through fine-tuning on benign and malicious prompts or structured negative examples emphasizing realistic injection patterns.These changes may affect refusal behavior, reasoning performance, and sensitivity to surface-level cues.Source: §B.1 · p. 11
• Prompt guards classify inputs before they reach the LLM, using lightweight classifiers, rules, or benchmark-trained models to label prompts safe or unsafe.Evaluated guards include ProtectAIv2, LakeraGuard, PromptGuard2, Deepset, and FMOPS; unlike fine-tuning defenses, they do not modify the LLM.Source: §B.1 · pp. 11–12

B.2.1 Datasets

• Position-stress evaluation draws benign reasoning tasks from GPQA, MMLU, and AIME 2024.The collection includes 100 GPQA Diamond questions, 1,204 MMLU validation questions, and all 30 AIME 2024 problems.Source: §B.2.1 · p. 12

B.2.2 H1: Position Stress Input Construction

• The position test compares independently presented questions with randomly paired, semantically unrelated questions concatenated in sequence, evaluating each response separately.A five-question MMLU ablation uses 173 sets and the Mistral base model and StrucQ-tuned variant.Source: §B.2.2 · pp. 12–13

B.2.3 Position Bias propmts H1

• The Llama and Mistral position prompts request step-by-step answers to two multiple-choice questions in a fixed answer format.The Mistral template adapts the SecAlign/StrucQ-style protocol to its instruction interface.Source: §B.2.3 · p. 13

B.3 Token Trigger Bias (H2)

• Token-trigger evaluation compares refusal rates on a benign validation mixture with rates on benign prompts containing three-word triggers.The mixture combines 971 benign WildGuard requests with approximately 2.8k benign PINT samples; the trigger subset has 113 prompts per trigger-count level.Source: §B.3 · p. 13
• Generative models are tested as binary injection classifiers, with refusals identified from refusal patterns; dedicated prompt guards are evaluated on the same baseline and trigger-stress conditions.The trigger-removal comparison distinguishes refusal rate with triggers from refusal rate after trigger tokens are removed.Source: §B.3 · pp. 13–14, §B.3.1 · p. 14

B.3.1 LLM Injection Detection Prompt

• The injection-classification prompt asks whether a user input is a prompt-injection attempt and requires a YES or NO response.The prompt frames the task as judging whether the input is safe to present to a chatbot.Source: §B.3.1 · p. 14, LLM Injection Detection Prompt · p. 14

B.4 Topic Generalization Set (H3)

• Topic generalization is tested on single benign reasoning tasks from GPQA, MMLU, and AIME, using accuracy and its absolute drop from the base model.The benchmarks cover diverse academic and professional domains outside much of the topical support of defense training data.Source: §B.4 · p. 14

B.5 Evaluation Metrics

• The evaluation defines refusal rate as rejection probability for benign inputs and measures utility with benign-task accuracy and its defended-versus-base change.Position effects compare rejection by task position, while trigger effects compare baseline and trigger-set refusal rates and their gap.Source: §B.5 · pp. 14–15

C.1 Extended Analysis of Main Results

• Higher attack rejection on prompt-injection benchmarks coincides with substantial accuracy losses on benign GPQA, MMLU, and AIME tasks.The authors warn that aggregate rejection metrics can obscure widespread benign damage.Source: §C.1 · p. 15
• Suffix-task refusals rise sharply, and rejection probability increases monotonically with task position, although the concatenated tasks are independently benign.The authors attribute this pattern to position-driven shortcut learning rather than semantic difficulty.Source: §C.1 · p. 15
• Trigger insertion increases refusals while matched non-trigger controls do not, isolating token identity as a factor in rejection behavior.The authors interpret this asymmetry as consistent with reliance on surface patterns rather than policy evaluation.Source: §C.1 · p. 15
• Accuracy degradation across GPQA, MMLU, and AIME indicates poor generalization beyond defense-training topics, despite the benchmarks containing no adversarial instructions.The authors describe the loss as an alignment cost on benign tasks from unfamiliar domains.Source: §C.1 · pp. 15–16

C.2 In-Depth Analysis

• Binary benign/malicious supervision leaves intent uncoded, allowing regularities such as position and token identity to become easy rejection predictors.The absence of benign counterexamples resembling attacks leaves safe inputs matching attack-like forms unpenalized when rejected.Source: §C.2 · p. 16
• SFT defenses also shorten outputs on benign reasoning benchmarks, reflecting premature generation termination or truncated reasoning chains.The passage links this broad suppression of complex generation paths to utility degradation.Source: §C.2 · p. 16
• Removing attack-associated trigger tokens from unchanged benign prompts sharply reduces prompt-guard refusals, isolating token patterns rather than policy violations as the driver.Source: §C.2 · p. 16
• Concatenated benign questions produce a sharp response-rate drop from question two onward in defended models, while base models remain stable; unanswered questions monotonically lower accuracy.The questions are independently sampled and lack adversarial structure; the passage attributes the pattern to a prefix-sensitive rejection heuristic.Source: §C.2 · p. 16