PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Indirect Prompt Injection Attacks Be Detected and Removed?

Yulin Chen, Haoran Li, Yuan Sui, Yufei He, Yue Liu, Yangqiu Song, Bryan Hooi

arXiv:2502.16580v5cs.CR

TL;DR

Indirect prompt injection detection and post-detection removal remain less studied than direct-attack detection. The paper constructs benchmarks and training data, evaluates existing and trained detectors, and compares segmentation and extraction removal. Existing models struggle, trained models perform satisfactorily, segmentation is stronger overall, and extraction is especially effective for tail-position injections, while generalization and robustness remain limited.

  • Problem

    Most detection work targets direct prompt injection, with limited evidence for indirect attacks and for removing injected instructions after detection.

  • Method

    The paper constructs indirect-attack benchmarks and training datasets, evaluates existing and trained detection models, and tests segmentation and extraction removal methods.

  • Results

    Existing LLMs and open-source detectors struggle with indirect attacks, specifically trained models perform satisfactorily, segmentation removes more overall, and extraction removes at least 94.66% of tail injections with Qwen2-1.5B.

  • Takeaways & Limitations

    Indirect prompt-injection filtering is effective when detection and removal are combined, but removal effectiveness and position generalization remain areas for improvement.

  • Takeaways & Limitations

    The study does not evaluate direct prompt injection, and its simple removal methods are not entirely satisfactory.

Abstract

from arXiv · show

Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions, because of their instruction-following capabilities and inability to distinguish between the original input instructions and maliciously injected instructions. To defend against such attacks, recent studies have developed various detection mechanisms. If we restrict ourselves specifically to works which perform detection rather than direct defense, most of them focus on direct prompt injection attacks, while there are few works for the indirect scenario, where injected instructions are indirectly from external tools, such as a search engine. Moreover, current works mainly investigate injection detection methods and pay less attention to the post-processing method that aims to mitigate the injection after detection. In this paper, we investigate the feasibility of detecting and removing indirect prompt injection attacks, and we construct a benchmark dataset for evaluation. For detection, we assess the performance of existing LLMs and open-source detection models, and we further train detection models using our crafted training datasets. For removal, we evaluate two intuitive methods: (1) the segmentation removal method, which segments the injected document and removes parts containing injected instructions, and (2) the extraction removal method, which trains an extraction model to identify and remove injected instructions.

1 Introduction

Existing detection approaches mainly target direct prompt injection and leave indirect attacks and post-detection removal insufficiently explored. This paper evaluates indirect-attack detection and two methods for removing injected instructions.

  • Filtering approaches detect injected documents and remove instructions, but prior methods largely focus on direct attacks and leave removal underexplored.Smaller detection models reduce computational demands and prompt-crafting requirements.
  • The paper constructs an indirect prompt injection benchmark from QA documents and manually crafted injected instructions.It evaluates current LLMs and detection models, then creates additional training data for new detection models.
  • Segmentation removal divides injected documents into segments and discards segments classified as containing injected instructions.
  • Extraction removal trains a model to identify and remove injected content directly from documents.
  • Specifically trained models show satisfactory detection, while existing instructed LLMs and open-source detectors struggle with indirect attacks.
  • Segmentation and extraction remove some injected instructions, with segmentation performing better overall and extraction excelling for tail-position injections.

2 Related Work

Related work describes prompt injection as a significant challenge for broadly adopted LLMs and surveys defenses that reinforce instructions or delimit data content.

  • Prompt injection attacks are a critical challenge for LLMs that have been broadly adopted across diverse tasks.
  • Prior defenses append reminders to reinforce adherence to original instructions.
  • Other defenses use special tokens to delineate the data-content area.

3 Benchmark and Training Datasets

The paper builds benchmarks and training datasets for evaluating indirect prompt-injection detection, removal, and defense. Its data represent original instructions, documents, injected instructions, probes, and attack positions across QA-derived samples.

  • The benchmark evaluates detection, removal, and defense robustness against indirect prompt injection attacks.
  • Each benchmark sample is represented as (p, d, a, x, y), covering the original instruction, clean document, answer, injected instruction, and execution probe.
  • The benchmarks use 900 validation samples each from SQuAD and TriviaQA, with manually designed injected instructions and probes.
  • Training data combine SQuAD and TriviaQA documents with Stanford-Alpaca instructions to create clean and injected document pairs.
  • Detection training data contain 40% clean documents, 15% head injections, 30% middle injections, and 15% tail injections.
  • Extraction training places each injected instruction at head, middle, and tail positions, tripling each document-pair sample for positional coverage.
  • Detection uses true positive rate for injected documents and false positive rate for clean documents, while removal uses removal rate.

4 Detect and Remove Indirect Prompt Injection Attacks

The paper formulates indirect prompt-injection defense as document classification followed by removal, evaluating detection models and two removal strategies. Detection maps documents to injected or clean labels, while removal either filters segments or extracts and deletes the injected instruction.

  • 4 Detect and Remove Indirect Prompt Injection Attacks: Indirect attack detection classifies injected documents as 1 and clean documents as 0, while removal should produce a document free of the injected instruction.Injected documents are generated from clean documents and instructions placed at the head, middle, or tail.
  • 4.1 Attack Detection: The detection pipeline considers classification and generative models that derive logits from document hidden states before selecting the predicted label.Classification uses the first hidden state, whereas generative detection uses the last hidden state and the logits for “no” and “yes.”
  • 4.1 Attack Detection: The prediction is the class or token associated with the largest logit, and the paper trains detection models on crafted data using cross-entropy loss.The loss distinguishes binary classification from generative detection, whose class count equals the vocabulary size.
  • 4.2 Attack Removal: The two removal strategies are evaluated as methods for deleting injected instructions from documents after detection.The paper explicitly investigates segmentation removal and extraction removal for injected documents.
  • 4.2 Attack Removal: Segmentation removal divides an injected document into sentence-level segments, classifies each segment, and combines the segments predicted to be clean.The same detection model is used for segment classification and document-level attack detection.
  • 4.2 Attack Removal: Extraction removal trains a model to identify and remove the injected instruction, including its start and end positions within the document.The extraction model is trained to extract the injected instruction completely before removal.
  • 4.2 Attack Removal: The extraction loss combines language-modeling terms with additional terms emphasizing accurate identification of the injected instruction’s boundaries.The model parameters are optimized using the extraction model’s training objective.
  • 4.2 Attack Removal: After extraction, the longest common substring between the extracted instruction and candidate document is identified and removed.This connects instruction extraction to the final document-processing operation.

5 Experiments

The experiments evaluate attack and defense baselines, detection models, removal methods, and unified filtering for indirect prompt injection. Specifically trained models detect attacks better than existing models, but performance depends on domain, fluency, injection position, and removal strategy.

  • Experimental setup: The evaluation compares five attack baselines and three defense baselines alongside trained detection and removal models.Attacks include Naive, Ignore, Escape, Fakecom, and Combined; defense baselines include Sandwich, Instructional, and StruQ.
  • Detection performance: 78.74% average accuracy for Llama3-8B-Instruct and 42.54% for Qwen2-7B-Instruct on Inj-SQuAD show that instructed LLMs struggle with indirect injection detection.Llama-Guard reaches at most 39.11% accuracy, while Prompt-Guard performance varies by attack and position.
  • Detection performance: 97.20% average accuracy for Qwen2-1.5B and 99.12% for DeBERTa on Inj-TriviaQA show that crafted-data training substantially improves detection.These models generalize from Naive-attack training to other attack types, although over-defense still occurs.
  • Over-defense: Over-defense is minimal in-domain but emerges out-of-domain, and lowering the injection rate trades reduced over-defense for lower defense performance.For Qwen2-0.5B, reducing the injection rate from 0.2 to 0.1 increased the over-defense rate, so injection-rate reduction is not optimal alone.
  • Over-defense and position effects: Stronger models and more fluent documents are less prone to out-of-domain over-defense, while single-position training generalizes poorly to other injection positions.Middle-position training generalizes better than head- or tail-only training, but robust detection still requires all positions during training.
  • Removal performance: Segmentation removal performs better overall, whereas extraction removal is strongest for tail injections, reaching at least 94.66% removal with Qwen2-1.5B.Extraction removal struggles particularly with head and middle Fakecom and Combined attacks, with a maximum removal rate of 67.77%.
  • Unified defense: Unified filtering generally outperforms prior prompt-engineering and fine-tuning defenses, although both approaches trained only on Naive attacks generalize poorly to Fakecom attacks.The filtering system uses DeBERTa for segmentation removal and Qwen2-1.5B for extraction removal.
  • Clean-data impact: Despite detection over-defense, subsequent removal methods rarely eliminate essential clean-document information, preserving document usefulness.The evaluation examines the impact of over-defense and removal on the original QA task performance.

6 Conclusion

The paper studies detection and removal of indirect prompt injection attacks using evaluation benchmarks and training datasets. It finds that existing models struggle to detect these attacks reliably, while removal methods remain imperfect.

  • The paper constructs two evaluation benchmarks containing injected instructions designed for different purposes.
  • Existing models struggle to reliably detect indirect prompt injection attacks, while training faces over-defense and position-generalization challenges.
  • The study crafts training datasets to investigate detection-model training challenges.
  • Removal methods can eliminate some injected instructions, but their effectiveness still leaves room for improvement.

Limitations

The paper evaluates two simple removal methods for indirect prompt injection, but their performance is not entirely satisfactory. Its assessment excludes direct prompt injection.

  • The two removal methods are simple and easy to implement, but their performance is not entirely satisfactory.
  • The assessment covers indirect prompt injection and does not consider direct prompt injection.

Ethical Considerations

The work is presented as an empirical study of defenses against prompt injection attacks. Its benchmark and training data use existing datasets and crafted instructions that are not harmful or poisonous.

  • The work primarily conducts empirical studies about defending against prompt injection attacks.
  • The benchmark and training data combine existing datasets with crafted injected instructions that are not harmful or poisonous.
  • The authors state that this construction introduces no new safety risks concerning unsafe data samples.

A.1 Implementation Details

The experiments use PyTorch on a single NVIDIA H100-96G GPU, with fixed training and generation settings. Training uses one epoch and a learning rate of 1e-5.

  • The experiments use PyTorch 2.1.0 on a single NVIDIA H100-96G GPU.
  • Training uses a learning rate of 1e-5, one epoch, and a maximum length of 1280 with DeepSpeed.
  • Generation disables sampling and allows up to 256 new tokens, with max_length set to 8192.

A.2 Attack Baselines

The paper evaluates multiple indirect prompt-injection attack and defense strategies, then assesses detection, removal, model utility, and benchmark performance across attack categories.

  • Attack methods: Naive, ignore, escape-separation, fake-completion, and combined attacks modify document content to manipulate the model's instruction following.The appendix also lists sandwich, instructional, and StruQ defenses as attack-baseline or mitigation strategies.
  • Evaluation process: The evaluation generates an injected document from clean content, an injected instruction, an attack method, and an injection position.The process represents attack generation as d_inj = Atk(d, x, pos).
  • Evaluation process: Detection methods are expected to distinguish injected from clean documents, while removal methods should recover the clean document after processing.The processed document is evaluated as d_pro, with the intended transformation defined by the removal method.
  • Evaluation process: Final defense evaluation combines the original instruction with the processed document and checks whether the attack-success indicator appears in the model response.This simulates the indirect attack scenario after detection and removal.
  • Evaluation metrics: Table 4 measures model utility across defense methods using accuracy, with all results reported as percentages.The appendix also reports detection performance using false positive rate for clean documents and true positive rate for injected documents.
  • Evaluation metrics: Figure 6 reports minimal true positive rates for models trained with attacks at one injection position and distinguishes training position from evaluation position.Qwen2-ID denotes in-domain evaluation of the trained Qwen2-0.5B model, while Head(T) and Head(D) refer to head-position training and detection.
Loading 2502.16580v5…