PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models

Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, Fangzhao Wu

arXiv:2312.14197v4cs.CLcs.AI

TL;DR

Indirect prompt injection attacks exploit malicious instructions embedded in third-party content, while comprehensive mitigation evidence remains limited. This paper introduces BIPIA to benchmark the threat, analyzes its causes, and evaluates black-box and white-box defenses, finding near-zero ASR with the white-box method while preserving output quality.

  • Problem

    Indirect prompt injection attacks embed malicious instructions in external content that can cause harmful or misleading LLM responses, while comprehensive benchmarks for analyzing and mitigating them remain lacking.

  • Method

    The paper introduces BIPIA and evaluates boundary-awareness, explicit-reminder, and adversarial-training defenses in black-box and white-box settings.

  • Results

    The white-box defense decreases ASR to nearly zero while preserving fine-tuned LLM output quality; black-box defenses significantly reduce ASR with minimal performance impact.

  • Takeaways & Limitations

    BIPIA provides comprehensive coverage for evaluating indirect prompt injection attacks, while the evaluated defenses offer mitigation without substantial loss on benign and general tasks.

  • Takeaways & Limitations

    The authors caution developers against relying on these defenses without careful testing and red teaming for specific end-to-end deployments.

Abstract

from arXiv · show

The integration of large language models with external content has enabled applications such as Microsoft Copilot but also introduced vulnerabilities to indirect prompt injection attacks. In these attacks, malicious instructions embedded within external content can manipulate LLM outputs, causing deviations from user expectations. To address this critical yet under-explored issue, we introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Using BIPIA, we evaluate existing LLMs and find them universally vulnerable. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. Based on these findings, we propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings. Extensive experiments demonstrate that our black-box defense provides substantial mitigation, while our white-box defense reduces the attack success rate to near-zero levels, all while preserving the output quality of LLMs. We hope this work inspires further research into securing LLM applications and fostering their safe and reliable use.

1 Introduction

The paper introduces BIPIA to benchmark indirect prompt injection attacks, analyzes why LLMs follow malicious instructions in external content, and evaluates defenses designed to reduce attacks while preserving performance.

  • Threat: Indirect prompt injection attacks embed malicious instructions in external content, causing applications to produce harmful, misleading, or inappropriate responses.The example describes GPT-4 promoting fake antivirus software after processing malicious external instructions.
  • Benchmark and motivation: BIPIA benchmarks LLM vulnerability to indirect prompt injection attacks across application scenarios and attacker goals.The benchmark evaluates five application scenarios and 250 attacker goals.
  • Findings: All 25 evaluated LLMs show varying susceptibility to indirect prompt injection attacks, including GPT-3.5-turbo and GPT-4.The paper reports relatively higher vulnerability for these widely used models despite their strong capabilities.
  • Root causes: Two challenges facilitate these attacks: distinguishing informational context from actionable instructions and avoiding execution of instructions embedded in external content.These challenges motivate the paper’s boundary-awareness and explicit-reminder defenses.
  • Defenses: Black-box defenses reduce attack success rates, while the white-box defense substantially decreases attacks with little adverse impact on output quality.Black-box methods use prompt-based techniques; white-box methods modify the model and employ adversarial training.

2 Problem Definition

The problem definition describes how an LLM-integrated application combines user instructions with retrieved external content and evaluates defenses by balancing robustness against attack success with regular-task performance.

  • Application pipeline: An application retrieves external content and combines it with a user instruction through a predefined prompt template to form the LLM prompt.The formal construction is P = Combine(T, C, f(I)).
  • Defense goals: Defense aims to reduce the attack success rate while preserving performance on regular tasks and user-expected outputs.The two goals are robustness and performance.

3 Threat Model

The threat model specifies attackers’ goals, knowledge, and capabilities when targeting LLM-integrated applications through malicious instructions in external content.

  • Threat model: Attackers seek to inject malicious instructions that produce irrelevant responses or conduct targeted attacks.Their assumed knowledge includes public details of the target LLM and possibly the application when it is open-sourced.

4 BIPIA Dataset Construction

BIPIA is constructed to evaluate indirect prompt injection robustness across representative application tasks, attack types, and attack positions within external content.

  • Construction factors: BIPIA is organized around application task, attack type, and attack position within external content.These three factors define the dataset construction.
  • Application tasks: The benchmark covers email QA, web QA, table QA, summarization, and code QA scenarios drawn from real-world or established datasets.Examples include OpenAI Evals emails, NewsQA, WikiTableQuestions, XSum, and collected Python code samples.
  • Attack types: The dataset includes 30 text attack types and 20 code attack types spanning task-irrelevant, task-relevant, targeted, passive, and active attacks.Each attack type contains five specific malicious instructions.

5 Evaluation LLMs under Attacks

BIPIA evaluation finds broad vulnerability to indirect prompt injection across LLMs, with susceptibility shaped by model capability, attack type, application task, and instruction position. More capable models are more vulnerable on text tasks, while attack success also varies with task relevance, code-attack behavior, and prompt placement.

  • Overall vulnerability: All evaluated LLMs exhibit some vulnerability to indirect prompt injection attacks, with GPT-4 and GPT-3.5 showing relatively higher vulnerability.The evaluation covers a wide array of existing LLMs and reports varying susceptibility across attacks and application tasks.
  • Model capability: Higher model capability is positively associated with ASR for all attacks and text attacks, but not code attacks.Figure 2 reports Pearson coefficients of 0.6423 (p< 0.001) for all tasks, 0.6635 (p< 0.001) for text tasks, and -0.0254 for code tasks.
  • Application tasks: Summarization has higher ASR than table QA, email QA, and web QA, while code QA exceeds those three text QA tasks.The paper does not directly compare code QA with the text attacks because they are distinct targeted attack settings.
  • Attack types: Task-relevant text attacks and targeted attacks achieve higher ASRs than task-irrelevant text attacks, especially for GPT-4 and GPT-3.5-turbo.The authors suggest attention to task-relevant information and compatibility with the original task may contribute to this pattern.
  • Attack types: GPT-4 has notably lower ASR for active code attacks than passive code attacks, unlike the similar passive-active trend observed for other evaluated models.The authors relate this difference to GPT-4’s advanced code comprehension and ability to identify some malicious code snippets.
  • Instruction position: Attack instructions placed at the end of external content produce the highest ASR, followed by placement at the beginning and middle.The authors attribute this ordering to a possible position bias learned from training examples in which instructions commonly appear at the end.

6 Defense Methodology

The paper attributes indirect prompt injection success to confusion between external information and actionable instructions, then develops boundary-awareness and explicit-reminder defenses for black-box and white-box settings.

  • Two root causes are difficulty distinguishing informational context from actionable instructions and failure to avoid executing instructions embedded in external content.
  • Boundary awareness separates external content from user instructions, while explicit reminder tells the model not to execute embedded instructions.
  • Black-box Defense: Black-box defenses use prompt-learning methods, including multi-turn dialogue and in-context learning, without requiring access to model parameters.
  • Black-box Defense: Multi-turn dialogue moves potentially malicious third-party content to a previous conversation turn and places instructions in the current turn.
  • Black-box Defense: In-context learning provides examples of responses unaffected by malicious external instructions before presenting a new task.
  • White-box Defense: White-box defense modifies the model through adversarial training so it learns to ignore instructions in external content.
  • White-box Defense: The white-box method constructs training prompts with malicious external instructions and collects benign responses unaffected by them.
  • White-box Defense: Special tokens <data> and </data> mark external-content boundaries, with added embeddings incorporated into the model’s embedding matrix.

7 Experiments

The experiments evaluate black-box and white-box defenses across several LLMs, measuring attack success and whether defenses preserve benign-task performance. Both defense types reduce ASR, with white-box methods approaching complete mitigation; ablations indicate boundary awareness is especially important.

  • Dataset and Experimental Settings: The study evaluates black-box defenses on four LLMs and white-box defenses on Vicuna-7B and Vicuna-13B, using BIPIA and benign-performance tests.Benign evaluations include BIPIA-Clean with ROUGE and MT-Bench for general helpfulness.
  • Performance Comparison: Black-box defenses substantially reduce ASR across GPT-4, GPT-3.5-Turbo, Vicuna-7B, and Vicuna-13B while generally preserving ROUGE performance.In-context learning is the exception, showing a performance difference from the original model.
  • Performance Comparison: More powerful LLMs generally have higher ASRs, although their ASRs can decrease below those of weaker models on shorter external-content tasks.The authors attribute this pattern to higher base ASR and better following of explicit reminders or distinguishing boundaries in shorter tasks.
  • Performance Comparison: White-box defenses reduce ASR to close to 0, approximately 10 times below the original ASR, while at least one response construction method preserves ROUGE and MT-Bench capability scores.The results indicate near-complete protection without compromising model performance.
  • Ablation Study: Removing either explicit reminder or boundary awareness increases black-box ASR, while neither removal significantly changes ROUGE on the original task.This supports the effectiveness of both components without indicating a utility loss.
  • Ablation Study: Removing boundary awareness affects ASR more than removing explicit reminder in both black-box and white-box settings.White-box fine-tuning preserves performance on the fine-tuning task without attack but may affect general-task performance.

8 Conclusion

The paper introduces BIPIA to study indirect prompt injection and attributes attack success to failures in separating context from instructions and avoiding embedded commands. It proposes black-box and white-box defenses, with near-zero ASR reported for the white-box approach while preserving output quality.

  • 8 Conclusion: BIPIA is introduced as the first benchmark for indirect prompt injection attacks, with coverage across varied tasks and attack types.The benchmark is intended to support comprehensive evaluation of vulnerabilities and defenses.
  • 8 Conclusion: The paper proposes that attacks succeed because LLMs struggle to distinguish informational context from actionable instructions and to avoid executing embedded external-content instructions.These are presented as key conjectures about the root causes of attack success.
  • 8 Conclusion: Black-box defenses use prompt-learning techniques, while white-box defenses mark external content with special tokens and apply adversarial training.Black-box methods include in-context learning, border strings, multi-turn dialogue, and datamarking.
  • 8 Conclusion: Black-box defenses effectively reduce ASR but do not make LLMs fully robust, whereas white-box defenses decrease ASR to nearly zero while preserving fine-tuned-model output quality.The conclusion contrasts substantial mitigation with the stronger robustness reported for white-box fine-tuning.
  • 8 Conclusion: The authors expect this work to stimulate research toward more secure and reliable LLM applications.This is stated as the paper’s broader research motivation.

Ethical Consideration

The ethical discussion focuses on improving safety when LLMs use third-party content while recognizing that the benchmark and defenses have risk boundaries. The authors exclude attacks that could harm property or health and caution against relying on defenses without application-specific testing and red teaming.

  • Ethical Consideration: The work focuses on enhancing LLM safety and reliability when models are integrated with third-party content.The stated focus concerns safer use of LLM-integrated applications.
  • Ethical Consideration: Manual review excludes attacks that could harm personal property and health, reducing the harmfulness of the benchmark’s attacks.This mitigation addresses the risk that the benchmark could raise awareness of attacks for malicious misuse.
  • Ethical Consideration: Despite reported efficacy and low implementation overhead, developers are cautioned against overreliance on the defenses without careful end-to-end testing and red teaming.The caution applies specifically to deployment in particular applications.

B Additional Experimental Settings

The appendix points to tables that provide the detailed category information for the test and training attacks used in the experiments.

  • B Additional Experimental Settings: Table 5 contains detailed category information for test attacks, while Table 6 contains the corresponding information for training attacks.The passage distinguishes the attack-category documentation by evaluation split.

C.1 Hyper-parameter Analysis

The analysis examines in-context examples, response construction, and training steps as defense hyper-parameters. Response construction methods and sufficient training substantially reduce ASR while limiting benign-task performance impact.

  • The analysis studies the number of in-context examples, response construction methods, and training steps for defense performance.
  • Impact of the number of examples in the in-context learning: Adding different numbers of in-context examples can reduce ASR, but text tasks show no clear correlation between example count and ASR.The authors relate this pattern to diversity in external content and instructions.
  • Impact of different response construction methods: All three response construction methods reduce ASR to nearly 0, with GPT-4 performing best among them.GPT-4 has the least impact on ROUGE-1 for benign prompts, followed by Original LLM and BIPIA.
  • Impact of training steps: A significant ASR drop appears after approximately 100 training steps, while ROUGE and MT-Bench capability scores stabilize after 500 steps.
Loading 2312.14197v4…