PaperlayerView exact revision on arXiv

Source-linked AI summary

PISanitizer: Preventing Prompt Injection to Long-Context LLMs via Prompt Sanitization

Runpeng Geng, Yanting Wang, Chenlong Yin, Minhao Cheng, Ying Chen, Jinyuan Jia

arXiv:2511.10720v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Long-context prompt injection is challenging because malicious instructions occupy only a small part of the context. PISanitizer sanitizes influential tokens before backend generation, and the paper reports near-zero attack success rates alongside maintained utility and robustness to strong attacks.

  • Problem

    Long-context LLMs can be misled by injected instructions, which are difficult to detect because they occupy only a small portion of the context.

    Source: §1 · p. 1

  • Method

    PISanitizer prompts an LLM to follow instructions in a context and sanitizes tokens that drive that behavior before a backend LLM generates the response.

    Source: §1 · pp. 2–3

  • Results

    PISanitizer reduced attack success rates close to 0, maintained utility, outperformed state-of-the-art baselines, and resisted optimization-based and strong adaptive attacks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    PISanitizer provides a sanitization-based defense for long-context scenarios, with evaluations showing effective prompt-injection defense and maintained utility.

    Source: §1 · p. 3, §7 · p. 20

    PISanitizer may be less effective against a potential weak attack whose instruction receives little or no attention but is still followed, if possible.

    Source: §6.1 · p. 19

Abstract

from arXiv · show

Long context LLMs are vulnerable to prompt injection, where an attacker can inject an instruction in a long context to induce an LLM to generate an attacker-desired output. Existing prompt injection defenses are designed for short contexts. When extended to long-context scenarios, they have limited effectiveness. The reason is that an injected instruction constitutes only a very small portion of a long context, making the defense very challenging. In this work, we propose PISanitizer, which first pinpoints and sanitizes potential injected tokens (if any) in a context before letting a backend LLM generate a response, thereby eliminating the influence of the injected instruction. To sanitize injected tokens, PISanitizer builds on two observations: (1) prompt injection attacks essentially craft an instruction that compels an LLM to follow it, and (2) LLMs intrinsically leverage the attention mechanism to focus on crucial input tokens for output generation. Guided by these two observations, we first intentionally let an LLM follow arbitrary instructions in a context and then sanitize tokens receiving high attention that drive the instruction-following behavior of the LLM. By design, PISanitizer presents a dilemma for an attacker: the more effectively an injected instruction compels an LLM to follow it, the more likely it is to be sanitized by PISanitizer. Our extensive evaluation shows that PISanitizer can successfully prevent prompt injection, maintain utility, outperform existing defenses, is efficient, and is robust to optimization-based and strong adaptive attacks. The code is available at https://github.com/sleeepeer/PISanitizer.

1 Introduction

Long-context prompt injections are difficult to detect because they occupy little of the context, while existing defenses have limited effectiveness in this setting. PISanitizer sanitizes tokens that drive instruction-following before a backend LLM responds, and its evaluation reports effectiveness, utility, efficiency, and robustness.

1 Introduction

  • Injected instructions are difficult to detect in long contexts because they constitute only a small portion, while existing defenses have limited effectiveness in this setting.The introduction notes limitations of prevention-, detection-, and attribution-based approaches for long contexts.Source: §1 · pp. 1–2
  • PISanitizer pinpoints and sanitizes injected tokens before a backend LLM generates an output from the sanitized context.It aims to remove injected tokens, remain efficient, and maintain utility for the target task.Source: §1 · p. 2
  • PISanitizer prompts an LLM to follow instructions in the context, then sanitizes tokens receiving high attention from the generated output token.The first generated token’s attention weights are used to identify influential tokens, avoiding generation of a full output for sanitization.Source: §1 · pp. 2–3
  • Considering attention weights across consecutive tokens captures the collective influence of malicious instructions and improves sanitization accuracy and robustness.The approach is motivated by the tendency of malicious-instruction tokens to occur consecutively.Source: §1 · p. 3
  • PISanitizer creates an attacker dilemma: injected instructions that more effectively compel instruction-following are more likely to be sanitized.This design targets injected tokens crafted by strong prompt injection attacks.Source: §1 · p. 3
  • PISanitizer reduced attack success rates close to 0, maintained target-task utility, outperformed state-of-the-art baselines, and resisted optimization-based and strong adaptive attacks.Sanitizing a context with thousands of tokens takes around 1.8 seconds.Source: §1 · p. 3

2 Background and Related Work

The background defines the long-context task and reviews heuristic and optimization-based injection attacks. It compares prevention, detection, and attribution defenses, then frames PISanitizer as context sanitization that must interpret semantic intent.

2.1 Long-context LLMs

  • A backend LLM performs a target task by following a target instruction while using a target context as input.The context supplies task data, such as retrieved webpages or documents.Source: §2.1 · p. 3

2.2 Prompt Injection Attacks

  • Prompt injections combine an injected instruction with a separator; attacks include heuristic strategies and optimized suffixes that increase the chance the LLM follows the instruction.Heuristic examples include Naive Attack and Combined Attack, while optimization-based attacks use gradient methods such as GCG.Source: §2.2 · p. 4

2.3 Prompt Injection Defenses

  • Prevention defenses try to preserve the target task, while detection and attribution defenses identify contaminated contexts and locate injected text; these approaches have reported limitations in effectiveness, long-context detection, or dependence on detectors.Fine-tuning defenses include StruQ and Meta-SecAlign; policy-based defenses face task-coverage and policy-specification challenges, and inference-time scaling can be inefficient.Source: §2.3 · pp. 4–5
  • Detection methods such as DataSentinel test whether a detection LLM follows an injected instruction, but the reviewed state-of-the-art methods do not accurately detect injected prompts in long contexts.Attribution methods locate injected prompts after contamination is detected and thus inherit detector limitations; some also achieve sub-optimal performance.Source: §2.3 · p. 5

2.4 Input Sanitization

  • PISanitizer applies input sanitization to remove or suppress malicious instructions before backend generation, but prompt-injection sanitization must interpret semantic intent because instruction and data are not clearly separated.Source: §2.4 · p. 5

3 Threat Model and Problem Formulation

The threat model considers an attacker seeking an attacker-desired response by injecting instructions into a long context, including under strong access and placement capabilities. Prompt sanitization removes potential injected tokens before backend generation while balancing effectiveness, efficiency, and task utility.

3.1 Prompt Injection to Long-Context LLMs

  • The attacker aims to place a malicious instruction in a long context so the backend LLM generates an attacker-desired output.Illustrated settings include manipulated answers, reviews, review summaries, and AI-generated code.Source: §3.1 · p. 6
  • The threat model assumes a strong attacker with white-box access to the backend model and system prompt, full context and target-instruction access, and arbitrary injection placement.Source: §3.1 · p. 6

3.2 Problem Setup for Prompt Sanitization

  • Prompt sanitization removes potential injected tokens from a target context before that context is sent to the backend LLM.The aim is to prevent an injected instruction in a contaminated context from influencing the backend output.Source: §3.2 · p. 6
  • Sanitization is evaluated against three goals: removing injected tokens, operating efficiently, and maintaining utility for the target task.Source: §3.2 · p. 6
  • A defender may be an application provider that filters contexts system-wide or an individual user who sanitizes a document before querying a backend LLM.Source: §3.2 · p. 6
  • The defender can use an LLM separate from the backend, such as a publicly available model for sanitization, and pass its sanitized context to the backend.Source: §3.2 · p. 6

4 Design of PISanitizer

PISanitizer intentionally elicits instruction-following behavior, then uses attention to identify and sanitize context tokens that drive it. Its detailed design aggregates token attention and detects likely injected spans from consecutive high-attention tokens.

4 Design of PISanitizer

  • PISanitizer intentionally elicits instruction-following behavior, then sanitizes context tokens receiving high attention from the generated output token.Generating a single output token captures the dominant influence of injected instructions while reducing computational cost.Source: §4 · p. 7, §4.1 · pp. 7–8
  • The more effectively an injected instruction compels the LLM to follow it, the more likely PISanitizer is to pinpoint and sanitize it.The sanitization instruction is unrelated to the target instruction, so the design aims to preserve clean tokens important to the target task.Source: §4 · p. 7

4.1 Detailed Design of PISanitizer

  • PISanitizer averages attention across heads within each layer, then takes the maximum across layers to amplify salient signals and suppress noise from less-informative layers.The paper reports that this aggregation improves sanitization effectiveness.Source: §4.1 · p. 8
  • It smooths attention scores, groups nearby peaks into spans, and treats the highest-scoring span as injected only when its score exceeds a threshold.The Savitzky–Golay filter smooths local fluctuations; nearby peaks are merged when their gap is below a predefined distance, and the threshold controls the utility-effectiveness tradeoff.Source: §4.1 · pp. 8–9
  • PISanitizer repeats sanitization until no tokens are removed or the maximum repetition count is reached.The experiments set that maximum to five repetitions.Source: §4.1 · p. 9

5 Evaluation

The evaluation examines PISanitizer’s ability to reduce prompt-injection success while preserving task utility across long-context tasks, models, injected tasks, defense components, and adaptive attacks. It also assesses performance in short-context and LLM-agent settings, including utility when there are no attacks.

5.1 Experimental Setup

  • The evaluation uses six LongBench datasets, randomly selecting 100 samples per dataset; unless noted, HotpotQA, Combined Attack with random-location injection, and Llama-3.1-8B-Instruct are defaults, with task-specific utility and ASR metrics.Source: §5.1 · pp. 9–10

5.2 Main Results

  • Under attacks, PISanitizer reduces ASR to nearly zero while preserving utility; token-level sanitization averages 0.80 precision, 0.90 recall, and 0.82 F1.Without attacks, it removes 0.81 tokens on average; some code-generation utility is not fully restored because occasional clean tokens are removed.Source: Table 1 · p. 11, §5.2 · p. 13
  • Using Llama-3.1-8B-Instruct for sanitization, PISanitizer consistently reduces ASR across open- and closed-source backends and four injected-task types.The evaluation includes three open-source and four closed-source backends, and target-answer, hacked-generation, classic-NLP, and general-instruction attacks.Source: Table 2 · p. 12, §5.2 · pp. 13–14, Table 3 · p. 13
  • PISanitizer outperforms baselines under Combined and GCG attacks, with comparable runtime and about 1.8s average context sanitization.PromptLocate is significantly slower on long contexts because it classifies many segments.Source: Table 4 · p. 14, §5.2 · pp. 14–15
  • Heuristic prevention, PromptArmor, and DataFilter are less effective; detection-based defenses and attribution combinations also fail through high false positives, false negatives, or utility loss.PromptLocate can reduce ASR to nearly zero but loses utility by removing clean tokens, while Meta-SecAlign has high ASR under GCG attacks.Source: §5.2 · pp. 14–15

5.3 Ablation Study

  • Jointly aggregating attention across consecutive tokens better balances utility and ASR than individual thresholding, and performance is relatively insensitive to θ, d, and ws.Low individual thresholds remove more injected tokens but harm utility; high thresholds preserve utility but allow some attacks to succeed.Source: Table 7 · p. 16, §5.3 · p. 16
  • Without a sanitization instruction, ASR rises; using the target instruction risks utility loss, while instruction-following sanitization prompts perform consistently well and noise-aware attention aggregation beats averaging.Target-task instructions can direct attention to clean tokens that are then mistakenly sanitized.Source: Table 8 · p. 16, §5.3 · p. 16, Table 9 · p. 16

5.4 Adaptive Attacks

  • Adaptive attacks test repeated injections, prompts that neutralize or override sanitization, tokens that disrupt peak grouping, and optimized suffixes that preserve attack success while suppressing attention.The optimized suffix has 50 tokens and is tuned with nano-GCG for up to 500 iterations against a combined effectiveness-and-stealth loss.Source: §5.4 · p. 17
  • The Optimization-II adaptive attack used nanoGCG for 500 iterations to optimize only attention suppression, making injected instructions more likely to bypass PISanitizer.The authors report that jointly optimizing attack effectiveness and attention suppression was challenging because the two objectives could pull the optimizer in different directions.Source: §5.4 · p. 17

5.5 Short Context and LLM Agent

  • PISanitizer consistently reduced attack success rates to nearly zero in short-context and LLM-agent evaluations while generally maintaining utility without attacks.The evaluations used TaskTracker, AlpacaFarm, SQuAD-v2, and Dolly for non-agent settings, and InjecAgent under the ENHANCED attack; utility was not evaluated for TaskTracker or InjecAgent.Source: Table 11 · p. 18, §5.5 · p. 18

5.4.1 Experimental Results

  • PISanitizer remained effective against adaptive attacks because its sanitization instruction induces instruction-following, drawing attention to injected instructions even when they oppose sanitization.The paper notes that reducing attention to instructional tokens an LLM follows can be challenging.Source: §5.4.1 · p. 18, Table 10 · p. 18

6 Discussion and Limitation

PISanitizer’s effectiveness is bounded by whether attacks rely on explicit instructions and whether followed instructions receive attention. It can also remove benign instructions that a target task needs.

6.1 Core Motivations of PISanitizer

  • PISanitizer fails against attacks that manipulate outputs without explicit instructions, such as knowledge-corruption attacks.The authors note that such attacks can be extremely challenging to defend against without extra information.Source: §6.1 · p. 19
  • Under Combined Attack, ASR fell from 0.75 to 0.0 and utility rose from 0.12 to 0.67 with PISanitizer, but utility did not reach the no-attack level.Without attacks, utility was 0.76 both with and without PISanitizer; injected non-instructional data still influenced utility in the short context.Source: §6.1 · p. 19
  • PISanitizer can be less effective when an injected instruction receives little or no attention but the LLM still follows it.The authors say adaptive optimization of injected prompts toward this condition is challenging, and suggest combining PISanitizer with Meta-SecAlign [17].Source: §6.1 · p. 19

6.2 Benign Instructions in a Context

  • Without attacks, utility fell from 0.82 to 0.71 with PISanitizer; 11 of 13 analyzed utility-impact cases contained essential benign instructions.The authors note that account legitimacy can be difficult to determine without additional information.Source: §6.2 · p. 19
  • On AgentDojo tool-call results with GPT-4o, ASR fell from 0.48 to 0.03 while utility rose from 0.40 to 0.67 under attack.The experiment follows [17] in sanitizing tool-call results.Source: §6.2 · p. 19
  • Potential mitigations include human approval for sensitive operations and automated validation, such as security policies [18], after untrusted instructions are pinpointed.PISanitizer can be used to pinpoint instructions in a context.Source: §6.2 · pp. 19–20

7 Conclusion and Future Work

PISanitizer sanitizes injected tokens before a backend LLM generates an output, addressing prompt injection in long-context scenarios. The authors report effective defense and identify extending the approach to multimodal LLMs as future work.

7 Conclusion and Future Work

  • The authors propose extending PISanitizer to multimodal LLMs as future work.They report that extensive evaluations demonstrate effective defense against prompt injection in long-context scenarios.Source: §7 · p. 20

A Details of Injected Tasks

The appendix describes four injected-task types used in experiments: target-answer generation, hacked generation, classic NLP tasks, and general-purpose instructions.

A Details of Injected Tasks

  • Target-answer generation uses attacker-chosen incorrect answers for QA, code generation, and retrieval, and a refusal-to-summarize answer for summarization.GPT-4o generates related but incorrect answers for Qasper, HotpotQA, LCC, and PassageRetrieval; GovReport and MultiNews use “Sorry, your context is too long to summarize.”Source: A Details of Injected Tasks · pp. 23–24
  • Hacked generation asks the LLM to print exactly “Hacked!”.The injected task follows Chen et al..Source: A Details of Injected Tasks · p. 24
  • Classic NLP injected tasks cover five datasets, with 20 randomly selected samples per dataset for 100 tasks total.The datasets are RTE, SST2, SMS-Spam, MRPC, and HSOL; their labels support ASR calculation.Source: A Details of Injected Tasks · p. 24
  • General-instruction attacks use 100 randomly selected instructions from the SEP benchmark’s 9.1K daily-task instructions.The appendix example adds a priority question about the river running through London.Source: A Details of Injected Tasks · p. 24

B Details on Algorithm 1

Algorithm 1 generates a token under the sanitization instruction, uses its attention to identify a high-attention token group, and removes that group only when its score exceeds a threshold.

B Details on Algorithm 1

  • The algorithm generates a token, aggregates attention over heads and layers, smooths the scores, groups peaks, and removes the highest-scoring group only if it exceeds θ.Peaks below 0.005 are filtered, and nearby peaks are grouped according to distance d.Source: B Details on Algorithm 1 · pp. 24–25
  • Defaults are ws = 9 for contexts over 500 tokens and ws = 5 for shorter contexts, with d = 10 and θ = 0.01.Sanitization repeats until no tokens are removed or five repetitions are reached.Source: B Details on Algorithm 1 · pp. 24–25

C Sanitization Instructions

The appendix evaluates four sanitization instructions that place a context between an instruction to follow it and a response prompt. The default adds a constraint to do only what the user asks and provide no explanation.

C Sanitization Instructions

  • The experiments evaluate sanitization instructions and use sanitization instruction 1 as the default unless otherwise specified; instruction 1 says to follow the context, do only what the user asks, and provide no explanation.Source: C Sanitization Instructions · p. 25
  • Instruction 2 omits the default instruction’s closing constraint to do only what the user asks without explanation.It ends after the context with “Your response:”.Source: C Sanitization Instructions · p. 25
  • Instructions 3 and 4 frame the context as noisy and ask the model to find and follow its instruction.Both include the default closing constraint to do only what the user asks without explanation.Source: C Sanitization Instructions · p. 25

D Implementation Details of LLM-as-a-judge

For classic NLP task and general instruction, the authors use GPT-4o as a judge of whether a response follows an injected instruction. The judge is asked to assess instruction-following, not correctness, and answer YES or NO.

D Implementation Details of LLM-as-a-judge

E Implementation Details of Optimization-based Attacks

The optimization-based attack appends an optimized suffix to the injected instruction. The suffix is optimized with nano-GCG until attack success or 500 iterations, using an initialization that follows [48].

E Implementation Details of Optimization-based Attacks

F Additional Experimental Results

Additional experiments report that PISanitizer remains effective across context lengths and is relatively robust to changes in group distance and smoothing window size. Raising threshold θ trades higher attack success for lower utility under attack, while clean utility is maintained.

F Additional Experimental Results

  • PISanitizer is effective across different context lengths constructed from LongBench-E.Table 13 reports its effectiveness under different context lengths.Source: F Additional Experimental Results · p. 26, Table 13 · p. 27
  • Increasing threshold θ raises ASR and lowers utility under attack, and sufficiently high θ disables sanitization.At the highest thresholds, ASR reaches its maximum and utility its minimum, consistent with no defense under the Combined Attack in Table 1.Source: F Additional Experimental Results · p. 26
  • Clean utility is maintained without attack, while changes in group distance d and smoothing window size ws cause only minor ASR and utility differences.Figure 3 shows the impact of θ, d, and ws under the Combined Attack.Source: F Additional Experimental Results · p. 26, Figure 3 · p. 27
  • Table 14 characterizes PISanitizer as accurate in sanitizing injected tokens and excludes the No Attack setting because it contains none.Source: Table 14 · p. 28
Loading 2511.10720v1…