PaperlayerView exact revision on arXiv

Source-linked AI summary

Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction

Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Xiaoling Bai, Chi Fei, Yabo Li, Haozhe Ma, Yangqiu Song, Bryan Hooi

arXiv:2504.20472v2cs.CR

TL;DR

Prompt injection exploits LLMs’ difficulty distinguishing instructions from data, and suppressing compliance with injected instructions remains challenging. The paper prompts models to reference the instructions they execute and filters responses using those references. It reports better performance than prompt-engineering baselines, performance comparable to fine-tuning, and minimal impact on general performance.

  • Problem

    LLMs’ strong instruction-following and difficulty distinguishing instructions from data leave them vulnerable to injected instructions, while suppressing compliance with those instructions remains challenging.

    Source: §1 · pp. 1–2

  • Method

    The method prompts LLMs to include references to the instructions they execute, then filters out responses not associated with the original instruction.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    The method outperforms prompt-engineering and fine-tuning baselines against direct and indirect attacks, with minimal impact on general performance.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Instruction references provide a way to leverage LLMs’ instruction-following ability while filtering responses unrelated to the original instruction.

    Source: §1 · p. 2, §6 · p. 9

    The approach depends on carefully designed prompts and examples, and its prompt-engineering basis makes formal mathematical analysis difficult.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated impressive performance and have come to dominate the field of natural language processing (NLP) across various tasks. However, due to their strong instruction-following capabilities and inability to distinguish between instructions and data content, LLMs are vulnerable to prompt injection attacks. These attacks manipulate LLMs into deviating from the original input instructions and executing maliciously injected instructions within data content, such as web documents retrieved from search engines. Existing defense methods, including prompt-engineering and fine-tuning approaches, typically instruct models to follow the original input instructions while suppressing their tendencies to execute injected instructions. However, our experiments reveal that suppressing instruction-following tendencies is challenging. Through analyzing failure cases, we observe that although LLMs tend to respond to any recognized instructions, they are aware of which specific instructions they are executing and can correctly reference them within the original prompt. Motivated by these findings, we propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs. Our approach prompts LLMs to generate responses that include both answers and their corresponding instruction references. Based on these references, we filter out answers not associated with the original input instructions. Comprehensive experiments demonstrate that our method outperforms prompt-engineering baselines and achieves performance comparable to fine-tuning methods, reducing the attack success rate (ASR) to 0 percent in some scenarios. Moreover, our approach has minimal impact on overall utility.

1 Introduction

LLMs’ strong instruction-following and difficulty distinguishing instructions from data leave them vulnerable to prompt injection, while suppressing compliance with injected instructions remains challenging. This paper instead uses instruction references to filter responses, reporting stronger defenses than prompt-engineering baselines, performance comparable to fine-tuning methods, and minimal impact on general performance.

1 Introduction

  • LLMs’ strong instruction-following abilities and difficulty distinguishing instructions from data make them vulnerable to injected instructions that divert them from the original task.Injected instructions may appear in data such as web pages retrieved by search engines.Source: §1 · p. 1
  • Existing defenses commonly try to enforce the original instruction by suppressing compliance with injected instructions, but suppression remains challenging and can lead to ineffective defense.The approaches include prompt engineering and fine-tuning.Source: §1 · p. 2
  • The proposed defense leverages instruction-following ability by prompting LLMs to reference the instruction they execute, then filtering responses using those references.The motivation comes from attack cases where a model references and executes an injected instruction.Source: §1 · p. 2
  • The method outperforms prompt-engineering baselines, achieves performance comparable to fine-tuning methods, reduces ASR to nearly 0% in some scenarios, and has minimal impact on general performance.Evaluations cover direct and indirect attacks.Source: §1 · p. 2
  • The authors report extensive evaluations of the approach on both open-source and closed-source models.The evaluations include direct and indirect prompt injection attacks.Source: §1 · p. 2

2 Related Work

The related work describes attacks that exploit LLMs’ instruction-following tendencies and defenses that seek to reinforce authorized instructions or restrict execution. Figure 1 contrasts an injected-instruction attack with the paper’s reference-based filtering approach.

2.1 Prompt Injection Attacks

  • Prompt injection attacks exploit instruction-following tendencies through methods such as ignoring prompts, fake responses, and optimized suffixes.The discussion distinguishes prompt-engineering and fine-tuning attack approaches.Source: §2.1 · p. 2

2.2 Prompt Injection Defenses

  • Figure 1 contrasts a phishing-instruction attack that produces a phishing link with a defense that filters responses using instruction references.The displayed system prompt is illustrative; the actual prompt is more complex.Source: Figure 1 · p. 3
  • Existing defenses use reminders, delimiters, signatures, or fine-tuning to reinforce adherence to authorized instructions or filter unauthorized prompts.The section groups prior defenses into prompt-engineering and fine-tuning methods.Source: §2.2 · pp. 2–3

3 Threat Model

The threat model covers direct and indirect prompt injection, with attackers seeking responses to injected instructions while lacking access to system prompts or model parameters. It assumes limited system knowledge for prompt-engineering attacks but model knowledge for gradient-based attacks.

3 Threat Model

  • For both direct and indirect attacks, the attacker’s goal is to make the model include answers to injected instructions.The experiments focus on direct-attack goal hijacking; indirect attacks place instructions in external data sources.Source: §3 · p. 3
  • Attackers can modify data content but cannot modify the system prompt, model parameters, or other system components.Direct-attack inputs are treated as data, while indirect attacks rely on retrieved external content.Source: §3 · p. 3
  • For prompt-engineering attacks, attackers are assumed to lack system knowledge; for gradient-based attacks, they are assumed to know the applied models.The paper treats adaptive attacks requiring detailed system knowledge as highly impractical under the first assumption.Source: §3 · p. 4

4 Methodology

The method formalizes defense as prompting an LLM and filtering its response, then uses tagged instruction references to retain only the original instruction’s answer. It splits and tags content, guides structured response generation, and removes responses with nonmatching tags.

4.1 Problem Formulation

  • The problem formulation models injected content as benign text combined with an attacker instruction and defines defense as prompting an LLM followed by filtering its response.The stated goal is to design the defense and filtering functions so the final response does not contain an answer to the injected instruction.Source: §4.1 · p. 4

4.2 Defense with Instruction Reference

  • The defense prompts the LLM to reference each executed instruction, then filters instruction-response tuples to retain only the original instruction’s response.The method uses tags because a model may paraphrase rather than reproduce the original instruction exactly.Source: §4.2 · p. 4
  • Data is split into lines of at most K words and each line receives a tag, while the original instruction and data are placed in distinct areas.Tags identify instructions, not necessarily every line, because some lines contain no instructions.Source: §4.2 · p. 4
  • The prompt asks the model to identify an instruction’s tag, answer that instruction, and end with a special token in a tuple-based output format.Two in-context examples reinforce adherence to the requested response structure.Source: §4.2 · p. 5
  • Filtering keeps responses associated with the original instruction’s tag and removes the tags and original instruction from the final output.The pipeline discards responses associated with other tags.Source: §4.2 · pp. 4–5

5 Experiments

Across direct and indirect prompt-injection evaluations, the method outperforms prompt-engineering defenses and maintains low ASR, while retaining general-task utility. Experiments also examine larger and closed-source models, gradient-based attacks, ablations, and example responses.

5.1 Experimental Settings

  • The evaluation covers 208 direct-attack samples and 900 samples each for Inj-SQuAD and Inj-TriviaQA, across open- and closed-source models and security and utility metrics.ASR counts a response as a successful attack if it contains the injected instruction’s answer; utility is accuracy on QA and sentiment tasks.Source: §5.1 · p. 5

5.2 Baselines

  • The comparisons include five attack baselines and four training-free defense baselines, plus StruQ as the fine-tuning comparison.Attack baselines are Naive, Ignore, Escape, Fakecom, and Combined; defense baselines are Sandwich, Instructional, Reminder, and Spotlight.Source: §5.2 · p. 6

5.3 Main Results and Analysis

  • At least 19.71% better than prompt-engineering baselines across direct attacks, the method reaches maximum ASRs of 4.00% on Inj-SQuAD and 7.00% on Inj-TriviaQA.StruQ struggles with unknown attacks, while the method defends against all baseline attacks in the indirect setting.Source: §5.3 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • The method does not degrade QA performance and decreases sentiment-analysis accuracy by an average of 1.53%, versus 0.53% for Sandwich and 5.77% for StruQ.These general-performance evaluations apply the defense without attacks.Source: §5.3 · pp. 6–7, Table 3 · p. 8
  • On models exceeding 70B parameters, the method’s maximum ASR is 2.78%; it also achieves below-1% average ASR on GPT-4o and GPT-4.1.On GPT-4o-mini, ASR reaches 0.22%; increasing model size does not produce dramatic performance improvements.Source: §5.3 · p. 7, Table 4 · p. 8, Table 6 · p. 9
  • On Qwen2-7B-Instruct, ASR is 6.25% under GCG and 8.65% under AutoDAN, compared with StruQ’s 10.10% and 14.42%, respectively.These are direct-attack evaluations on AlpacaFarm; the attacks use model gradients to reverse-engineer adversarial suffixes.Source: §5.3 · p. 7, Table 5 · p. 8

5.4 Ablation Study

  • Window size has little effect on defense or utility; line-wise splitting alone does not explain the gains, while omitting in-context examples can cause correct answers to be filtered out.A dynamic reference tag mitigates the risk that an exposed static tag could be exploited, without reported loss of general performance.Source: §5.4 · pp. 7–8

5.5 Case Study

  • In the case study, the model follows the guidance in a Naive attack but executes an injected instruction split across tag areas; an Ignore prompt does not mislead it.The examples show both successful defense and a case where the split injected instruction is executed using a former tag.Source: §5.5 · pp. 8–9

6 Conclusion

The paper proposes a prompt-injection defense that has LLMs reference instructions, then filters responses whose references do not match the original input instructions.

6 Conclusion

  • The method prompts LLMs to include instruction references and filters responses whose references are not associated with the original input instructions.The intended result is a clean final output.Source: §6 · p. 9
  • The method outperforms prompt-engineering and fine-tuning baselines against various direct and indirect prompt injection attacks.Source: §6 · p. 9
  • The method has minimal impact on LLMs’ general performance.Source: §6 · p. 9

Limitations

The referencing strategy depends on carefully designed prompts and example outputs; dynamic tags retain a low matching risk, and prompt engineering makes formal mathematical analysis difficult.

Limitations

  • The referencing strategy depends on carefully designed prompts and example outputs to teach the model the correct reference structure.Source: Limitations · p. 9
  • An attacker-selected tag can still coincide with the dynamic reference tag, though the probability is low and can be reduced by enlarging the vocabulary.Source: Limitations · p. 9
  • Because the method uses prompt engineering, formal mathematical analysis is difficult, a limitation also shared by prior defense approaches (Chen et al., 2024b,c).Source: Limitations · p. 9

Ethical Considerations

The work is an empirical study of prompt-injection defenses, and its benchmark and training data use existing datasets with non-harmful crafted instructions.

Ethical Considerations

  • The authors affirm adherence to the ACM Code of Ethics and the ACL Code of Conduct.Source: Ethical Considerations · p. 9
  • The work primarily conducts empirical studies about defending against prompt injection attacks, and its source code will be made publicly available.Source: Ethical Considerations · p. 9
  • The benchmark and training data use existing datasets, and the crafted injected instructions are not harmful or poisonous.The authors state this introduces no new safety risks concerning unsafe data samples.Source: Ethical Considerations · p. 9

A Implementation Detail.

Experiments use PyTorch 2.1.0 on one NVIDIA H100 GPU, with deterministic generation settings and a line word number K of 32.

A Implementation Detail.

B.1 Attack Baselines

The attack baselines vary how injected instructions are inserted into or framed within the data content, from simple appending to combined tactics.

B.1 Attack Baselines

  • The Naive attack appends the injected instruction directly to the original data content.An example is identified in Table 13.Source: §B.1 · p. 13
  • The Ignore attack adds an ignoring instruction before placing the injected instruction in subsequent content (Perez and Ribeiro, 2022).An example is identified in Table 15.Source: §B.1 · p. 13
  • Escape-Separation creates new spaces or lines to trick the LLM, while Escape-Deletion uses special tokens to simulate a deletion command (Liu et al., 2024c).The implementation uses Escape-Separation; an example is identified in Table 14.Source: §B.1 · p. 13
  • Fake completion inserts a fake response to make the LLM believe the task is finished, then places the attacker's instruction in subsequent content.An example is identified in Table 16.Source: §B.1 · p. 13
  • The Combined attack combines the attack methods described above (Liu et al., 2024c).An example is identified in Table 17.Source: §B.1 · p. 13

B.2 Defense Baselines

The defense baselines include prompt strategies and a fine-tuning method. The section also provides examples and reports ablations and evaluation settings for the proposed defense.

B.2 Defense Baselines

  • Sandwich restates the original instruction at the end of content (san, 2023), while Instructional adds an attack warning after it (ins, 2023).Examples are identified in Tables 18 and 19.Source: §B.2 · p. 13, Table 18 · p. 21, Table 19 · p. 21
  • Reminder adds an instruction not to follow subsequent content (Yi et al., 2023), whereas Spotlight uses a special token to help mark injected instructions as data (Hines et al., 2024).Examples are identified in Tables 20 and 21.Source: §B.2 · p. 13, Table 20 · p. 21, Table 21 · p. 22
  • StruQ (Chen et al., 2024a) leverages adversarial training (M ˛adry et al., 2017) to align LLMs with original input instructions; this implementation uses the “Naive attack” as an adversarial example during training to evaluate defense generalization.Source: §B.2 · p. 13
  • Examples cover tagging and splitting, the response-generation prompt, full input, and reference-based filtering.Table 7 and Table 8 show tagging/splitting and the prompt; Figures 2 and 3 show the full input and response-reference examples.Source: Table 7 · p. 14, Table 8 · p. 14, Figure 2 · p. 15
  • Figure 4 reports no significant performance impact from changing the words-per-line window size, while removing in-context examples significantly harms general accuracy and guideline adherence.The ablation captions identify the window-size study and the no-examples comparison with No Defense.Source: Figure 4 · p. 17, Figure 5 · p. 17
  • Tables 9–11 describe ASR evaluations on Inj-TriviaQA or Inj-SQuAD, while Table 12 reports SQuAD QA accuracy across no attack and defense settings.The captions specify indirect evaluation for Tables 9 and 10.Source: Table 9 · p. 17, Table 10 · p. 17, Table 11 · p. 18

Related papers

Loading 2504.20472v2…