Source-linked AI summary
Defending Against Indirect Prompt Injection Attacks With Spotlighting
Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, Emre Kiciman
TL;DR
LLMs can confuse adversarial instructions embedded in external data with user commands, creating indirect prompt-injection risks. The paper introduces spotlighting, which transforms input and supplements the system prompt to signal provenance. Across GPT-family experiments, spotlighting substantially lowers attack success while generally preserving task performance, subject to encoding limitations on lower-capacity models.
Problem
LLMs process concatenated inputs as boundary-less token streams, making them vulnerable to mistaking malicious instructions in external data for user commands.
Method
Spotlighting combines input transformations with system-prompt instructions to make the provenance of untrusted text more salient to the model.
Results
Spotlighting significantly reduces attack success across models and tasks, while datamarking and encoding produce negligible detrimental impacts on task performance.
Takeaways & Limitations
Spotlighting offers a simple, broadly applicable prompt-engineering defense against indirect prompt injection across many tasks and models.
Takeaways & Limitations
Encoding can impair task performance on lower-capacity models and should be validated for each use case, with recommended use limited to highest-capacity models such as GPT-4.
Abstract
from arXiv · showhide
Large Language Models (LLMs), while powerful, are built and trained to process a single text input. In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. However, the LLM is unable to distinguish which sections of prompt belong to various input sources. Indirect prompt injection attacks take advantage of this vulnerability by embedding adversarial instructions into untrusted data being processed alongside user commands. Often, the LLM will mistake the adversarial instructions as user commands to be followed, creating a security vulnerability in the larger system. We introduce spotlighting, a family of prompt engineering techniques that can be used to improve LLMs' ability to distinguish among multiple sources of input. The key insight is to utilize transformations of an input to provide a reliable and continuous signal of its provenance. We evaluate spotlighting as a defense against indirect prompt injection attacks, and find that it is a robust defense that has minimal detrimental impact to underlying NLP tasks. Using GPT-family models, we find that spotlighting reduces the attack success rate from greater than {50}\% to below {2}\% in our experiments with minimal impact on task efficacy.
1. Introduction
LLMs’ boundary-less token streams make them unable to reliably distinguish valid instructions from untrusted external text, enabling indirect prompt injection attacks. The paper introduces spotlighting techniques to clarify token provenance and reports strong attack reduction with minimal task impact.
- Problem: Indirect prompt injection embeds malicious instructions in external data, allowing attackers to hijack users’ sessions through their credentials.The user may remain unaware of the attack and become an innocent bystander or victim.
- Problem: Boundary-less token streams prevent LLMs from reliably distinguishing system instructions from untrusted external inputs.This structural limitation makes code and data appear in the same undifferentiated stream.
- Approach: Spotlighting transforms input and updates the system prompt to help the model distinguish safe token blocks from unsafe ones.The paper examines delimiting, datamarking, and encoding as spotlighting instantiations.
- Findings: Across models and tasks, spotlighting significantly reduces attack success rates while datamarking and encoding have negligible detrimental effects on underlying NLP performance.The techniques are presented as simple to implement across many tasks and models.
2. Background and Related Work
The background distinguishes indirect prompt injection from direct jailbreak-style attacks and situates XPIA within the expanding risks of LLMs connected to external data and capabilities. Existing safety work includes alignment tuning, prompt engineering, and detection systems.
- LLM Safety Context: LLM instruction-following systems are increasingly used for automated tasks, making their reliability and safety critical.These systems support planning and reasoning in addition to text completion.
- Indirect Prompt Injection: Indirect prompt injection occurs when malicious instructions placed in external sources are mistaken for the user’s desired intent.External sources include websites, emails, and text messages, and the user is often the victim.
- Indirect Prompt Injection: XPIA risks increase as LLM systems gain plugins or broader access patterns, with attacks demonstrated against systems including Bing Chat and Bard.These systems process external web information alongside user input.
- Indirect Prompt Injection: Unlike direct user prompt injection, XPIA hides potentially malicious instructions within lengthy external documents and can turn benign instructions into attacks.The paper characterizes XPIA as encompassing the user-driven prompt-injection problem.
- Related Work: Existing safety approaches include alignment tuning, prompt engineering, and classifier-based detection systems.Alignment tuning incorporates desired and undesired responses into training objectives.
3. Spotlighting
Spotlighting makes input provenance more salient by combining input transformations with explicit system-prompt instructions. The paper presents delimiting, datamarking, and encoding, with datamarking replacing whitespace throughout the input and encoding applying algorithms such as base64 or ROT13.
- 3.1. Overview: Spotlighting addresses the instruction–data distinction by transforming input and adding system-prompt guidance about how that input should be treated.The approach targets indirect prompt-injection susceptibility at the prompt level.
- 3.2. Spotlighting via Delimiting: Delimiting marks the input boundary with special tokens placed before and after the content.The system prompt instructs the model not to obey instructions between those markers.
- 3.3. Spotlighting via Datamarking: Datamarking interleaves a special token throughout the input by replacing whitespace and alerts the model to the transformation.For example, whitespace can be replaced with the character ˆ throughout the document.
- 3.4. Spotlighting via Encoding: Encoding applies algorithms such as base64, ROT13, or binary so capable LLMs can implicitly decode transformed input during task performance.The system prompt describes the encoding and instructs the model to decode without changing its task.
4. Experimental Methodology
The evaluation measures indirect prompt-injection success across GPT-family models and tasks using a synthetic attack corpus, then compares baseline and system-instruction defenses. Baseline ASR varies by task and model, while added anti-injection instructions have only modest effects.
- Models and settings: The experiments use text-davinci-003, GPT-3.5Turbo, and GPT-4 black-box models at temperature 1.0.Varying temperature showed no notable impact on XPIA susceptibility.
- Attack corpus and metric: Attack Success Rate is the fraction of the total corpus producing successful prompt injections.
- Attack corpus and metric: 1000 synthetic documents embed keyword payloads instructing models to forget prior instructions and return a special keyword.The corpus enables straightforward identification of successful and unsuccessful attacks.
- Baseline evaluation: Baseline ASR varies across tasks and models, so relative differences are more informative than absolute values.GPT-4 tends to have the lowest ASRs in this evaluation.
- System-instruction baseline: Adding system instructions not to follow document instructions has only modest effects on attack success rates.The added instructions provide almost no benefit for GPT-3.5-Turbo and leave Text-003 with significantly high ASR.
5. RESULTS
Spotlighting substantially reduces indirect prompt-injection attack success, with encoding generally strongest, while datamarking preserves downstream NLP performance. Encoding nevertheless requires high-capacity models because it can impair task performance on GPT-3.5-Turbo.
- Spotlighting effectiveness: Delimiters reduced ASR by about half with GPT-3.5-Turbo, while defensive instructions alone had only a negligible impact.The baseline ASR with the test dataset was around 60%.
- Spotlighting effectiveness: Datamarking reduced ASR from approximately 50% to below 3% with GPT-3.5Turbo and from 40% to 0.00% with Text-003.The same reduction trend also appeared in document Q&A across three model types.
- Spotlighting effectiveness: Encoding produced the best ASR outcomes, reaching 0.0% in summarization and nearly 0.0% in Q&A with GPT-3.5-Turbo.Encoding outperformed datamarking across the reported summarization and Q&A tasks.
- Impact on language tasks: Datamarking had no detrimental impact on performance across SQuAD Q&A, IMDB Sentiment, SuperGLUE Word-in-Context, and SuperGLUE BoolQ.The transformation provided a cue for distinguishing text blocks without materially obscuring the input.
- Impact on language tasks: Encoding can impair task performance on GPT-3.5-Turbo, whereas GPT-4 handled encoded text consistently well.The authors recommend encoding only with high-capacity models and validating performance for each use case.
- Overall recommendations: The authors recommend at least datamarking generally and encoding with high-capacity LLMs because encoding was the most effective defense against XPIA.They also advise quantifying encoding’s impact on downstream tasks.
6. Discussion
The discussion frames indirect prompt injection as a signaling problem caused by boundary-less token streams, and interprets spotlighting as a way to reduce interference between trusted and untrusted text.
- 6. Discussion: The paper acknowledges that spotlighting appears effective but that its underlying mechanism is not yet clearly understood.The stated intuition is that more obvious token-block boundaries help models avoid dangerous instructions.
- 6. Discussion: In-band telecommunications separated voice and signaling frequencies to reduce accidental interference, but intentional signal mimicry remained a security problem.The paper uses phone phreaking to illustrate why separation alone may not resolve deliberate interference.
- 6. Discussion: LLMs treat system instructions and external data within the same unstructured token stream, preventing reliable separation between control and content.The paper characterizes this as analogous to in-band signaling, where control and user data share one channel.
- 6. Discussion: Spotlighting transformations may reduce token interference by moving transformed blocks into a different region of representation space while retaining a shared channel.This analogy links datamarking and encoding to multi-frequency transmission rather than to a fully separate communication channel.
- 6. Discussion: Out-of-band signaling provides a separate channel for signaling information, offering a conceptual direction for improving language-model security.The discussion presents this telecommunications development as inspiration for future language-model defenses.
7. Conclusion
The conclusion presents spotlighting as a family of provenance-preserving transformations for mitigating indirect prompt injection. Across tasks and models, it significantly reduces attack success while the methods involve trade-offs that require use-case-specific selection.
- 7. Conclusion: Spotlighting transforms input text to make provenance more salient while preserving semantic content and task performance.The paper instantiates the approach through delimiting, marking, and encoding.
- 7. Conclusion: The evaluated spotlighting techniques significantly reduce attack success rates across different tasks, models, and scenarios.The conclusion also discusses trade-offs and recommendations for choosing among transformation methods.
8. Appendix
The appendix examines few-shot examples as an alternative defense: they can sharply reduce ASR, but their apparent effectiveness is constrained by attack-knowledge coverage and evaluation leakage risks.
- 8. Appendix: A model that mentions the attack keyword while completing the task may notice the attack without actually following it, complicating strict ASR labeling.The appendix distinguishes this gray area from clear attack success and failure.
- 8. Appendix: Few-shot examples teach the model through in-context demonstrations how to avoid obeying instructions embedded in documents.The strategy modifies the system prompt with examples tailored to indirect prompt injection scenarios.
- 8. Appendix: Below 5% ASR was observed with GPT3.5-Turbo under both few-shot treatments, which differed only slightly in their included examples.Both treatments produced a dramatic decrease relative to simply warning the model about prompt injection.
- 8. Appendix: Figure 9 compares two few-shot treatments, FS1 and FS2, that differ slightly in the examples included in the prompt.The figure passage identifies few-shot examples as helpful for reducing Attack Success Rate, while warning that they require caution.
- 8. Appendix: Few-shot defenses may generalize imperfectly because examples reflect current knowledge of attack tactics and can leak labels into evaluation data.The authors therefore prefer spotlighting methods that target structural prompt vulnerabilities.