PaperlayerView exact revision on arXiv

Source-linked AI summary

FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks

Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z. Morley Mao, Muhao Chen, Chaowei Xiao

arXiv:2410.21492v2cs.CRcs.CL

TL;DR

Indirect prompt injection can exploit external text in tool-enabled LLM applications, while training-time defenses may be costly and existing test-time defenses have limited effectiveness against adaptive attacks. FATH uses hash-based authentication to label responses and filter those matching user instructions. The paper reports the lowest ASR across five attack methods, three injection tasks, and both Llama3 and GPT3.5 models.

  • Problem

    Indirect prompt injection can steer LLM-integrated applications through malicious instructions in external text, while training-time defenses can be costly and existing test-time defenses are insufficient against adaptive attacks.

    Source: §1 · pp. 1–2

  • Method

    FATH uses secure input formatting, a security policy, and hash-based authentication tags to label responses and verify which ones correspond to user instructions.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    FATH achieves the lowest ASR across all five Threat Modeling 1 attack methods and three injection tasks on both Llama3 and GPT3.5, with near 0% ASR reported.

    Source: §5.3 · p. 7

  • Takeaways & Limitations

    The authors conclude that FATH provides state-of-the-art defense performance among existing test-time methods for securing LLM-integrated applications.

    Source: §6 · p. 9

    FATH requires substantial manual effort to design defense prompts for each application, with templates differing between OpenPromptInjection+ and InjecAgent.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have been widely deployed as the backbone with additional tools and text information for real-world applications. However, integrating external information into LLM-integrated applications raises significant security concerns. Among these, prompt injection attacks are particularly threatening, where malicious instructions injected in the external text information can exploit LLMs to generate answers as the attackers desire. While both training-time and test-time defense methods have been developed to mitigate such attacks, the unaffordable training costs associated with training-time methods and the limited effectiveness of existing test-time methods make them impractical. This paper introduces a novel test-time defense strategy, named Formatting AuThentication with Hash-based tags (FATH). Unlike existing approaches that prevent LLMs from answering additional instructions in external text, our method implements an authentication system, requiring LLMs to answer all received instructions with a security policy and selectively filter out responses to user instructions as the final output. To achieve this, we utilize hash-based authentication tags to label each response, facilitating accurate identification of responses according to the user's instructions and improving the robustness against adaptive attacks. Comprehensive experiments demonstrate that our defense method can effectively defend against indirect prompt injection attacks, achieving state-of-the-art performance under Llama3 and GPT3.5 models across various attack methods. Our code is released at: https://github.com/Jayfeather1024/FATH

1 Introduction

LLM applications gain capabilities from tools and external text, but that information creates security risks when attackers inject instructions that divert models from users’ intentions. FATH proposes a test-time authentication defense and reports strong results against indirect prompt injection attacks.

1 Introduction

  • Indirect prompt injection lets attackers place malicious instructions in external text, potentially steering LLMs away from users’ intentions and exploiting tool-enabled applications.The paper cites an example of attackers using code interpreter and web access to record chat histories and send them out.Source: §1 · p. 1
  • Training-time defenses can be impractical when developers lack access to backbone models or cannot afford fine-tuning and repeated retraining.Source: §1 · p. 2
  • Existing test-time defenses are reported as insufficient against adaptive attacks, while boundary rules can be contradicted by injected instructions such as “ignore previous instructions.”Source: §1 · p. 2
  • FATH pairs user instructions with HMAC-based authentication and uses secure input formatting, a security policy, and verification to identify responses for the user.Applications proceed only when rule-based parsing finds a match with the key.Source: §1 · pp. 2–3
  • FATH achieves near 0% ASR on GPT3.5 for various attack methods and 0% ASR on Llama3 against optimization-based attacks.The paper also reports 0% ASR on both models in the InjecAgent tool-usage benchmark.Source: §1 · p. 3

2 Related Work

Related work distinguishes indirect prompt injection and two broad defense settings. The paper focuses on test-time defenses because they avoid training costs, while prior approaches have not been shown sufficiently effective against adaptive attacks.

2 Related Work

  • LLM-integrated applications use fine-tuning or in-context learning to connect models with tools, with the latter approach described as increasingly promising.Examples of in-context applications include ReAct (Yao et al., 2022), Mind2web (Deng et al., 2024), and AutoGPT.Source: §2 · p. 3
  • Indirect prompt injection manipulates external text rather than directly inserting malicious content into an entire input prompt, and is the paper’s focus.Source: §2 · p. 3
  • Training-time defenses incorporate adversarial examples during fine-tuning, but the process still carries substantial training costs.The cited examples include Chen et al. (2024) and Yi et al. (2023).Source: §2 · p. 3
  • Test-time defenses leave model parameters unknown, but prior approaches have not been proven sufficiently effective against adaptive attacks.Adaptive attacks are designed using information gained from specific defense strategies.Source: §2 · p. 3

3 Threat Modeling

The threat models share an attacker goal and restricted access to external text, but differ in knowledge of defenses. A separate optimization-based setting considers attackers with broader access and gradient-based attacks.

3 Threat Modeling

  • Attackers aim to manipulate LLM-integrated applications through indirect prompt injection, but can modify only external text and cannot access user instructions or formatting templates.The attacker sees text responses, not model parameters or output logits.Source: §3 · pp. 3–4
  • Threat Modeling 1 assumes attackers lack defense details and includes five attack methods: Naive Attack, Escape Characters, Context Ignoring, Fake Completion, and Combined Attack.Source: §3 · p. 4
  • Threat Modeling 2 gives attackers full defense knowledge, while the optimization-based worst case grants prompt and parameter access and uses gradients to optimize injected text.Dynamic authentication tags still vary during inference, even if attackers simulate them during optimization.Source: §3 · p. 4

4 FATH: Authentication-based Test-time Defense

FATH is a test-time defense that separates user instructions from external text, labels responses by authorization, and returns the response to the user instruction. Its design combines secure input formatting, a security policy, and rule-based verification.

4.1 Preliminary

  • An indirect prompt injection modifies external text with an injected instruction and optional injected text, aiming to make the application follow that instruction instead of the user's.The defense function applies a prompt template to the user instruction and potentially attacked text before the backbone LLM produces an output.Source: §4.1 · p. 4

4.2 Authentication System Design

  • FATH formats user instructions and external text separately, prompts the LLM to label responses, then parses its output to return the authorized response.The verification function extracts the response to the user instruction using TAG4.Source: §4.2 · pp. 4–5
  • FATH generates five hash-based authentication tags for distinct purposes; an Authorized Response answers user instructions, while an Unauthorized Response includes anything else.TAG4 is used to authenticate the Authorized Response during parsing.Source: §4.2 · pp. 4–5, Table 1 · p. 5
  • FATH combines high-level security instructions, in-context examples, and the formatted prompt before querying the backbone LLM.The examples are paired vanilla and injected cases, retrieved using instruction similarity.Source: §4.2 · p. 5

4.3 Example

  • FATH's prompt template may vary across tasks, requiring careful task-specific design to enhance performance.The paper points to OpenPromptInjection and InjecAgent prompt examples in the appendices.Source: §4.3 · pp. 5–6

5 Evaluation

The evaluation uses OpenPromptInjection+ and InjecAgent to test FATH across injection tasks, model backbones, and attack settings. Results report strong defense against standard, adaptive, and optimization-based attacks, while ablations show the importance of both authentication components.

5.1 Benchmarks

  • OpenPromptInjection+ broadens evaluation beyond seven specific tasks by using general user instructions and three injection-task categories: URL, QA, and CLF.The categories cover URL repetition, answering other questions, and classification injections.Source: §5.1 · p. 6
  • InjecAgent tests direct-harm attacks in tool-integrated agents, including actions such as unauthorized financial transactions and home-automation manipulation.The benchmark uses malicious instructions injected into external text extracted from ReAct tool results.Source: §5.1 · pp. 6–7

5.2 Experimental Settings

  • The evaluation uses Llama 3 and GPT-3.5, four established test-time baselines, multiple attack methods, and ASR plus an unaided-answer Judge Score.OpenPromptInjection+ uses 100 Stanford Alpaca examples per injection task; InjecAgent uses 510 direct-harm examples. Optimization-based attacks are run only on Llama3 because GPT-3.5 parameters are inaccessible.Source: §5.2 · p. 7

5.3 Results

  • FATH achieves the lowest ASR across five Threat Modeling 1 attacks, three injection tasks, and both Llama3 and GPT3.5, outperforming prior defenses.The authors also observe a small Judge Score decrease, possibly because authentication verification filters out reasoning contents.Source: §5.3 · p. 7, Table 2 · p. 8

5.4 Defense against Adaptive Attacks

  • Under Adaptive Attacks, FATH achieves 0% ASR on GPT-3.5 and InjecAgent, and significantly lowers ASR on Llama3 in OpenPromptInjection+.Adaptive Attacks outperform Combined Attacks, and results are reported for both benchmarks.Source: §5.4 · p. 8, Table 3 · p. 8

5.5 Robustness against Optimization-based Attacks as Worst Cases

  • Against optimization-based attacks on Llama3, FATH achieves 0% ASR, while the No Defense setting has 70% ASR.The attack optimizes prefixed strings using gradient information; FATH uses Expectation Over Time across five authentication-tag sets.Source: §5.5 · p. 9

5.6 Ablation Studies

  • Removing either Authentication Tags or Security Policy degrades defense relative to FATH, especially under Adaptive Attack; removing Security Policy raises ASR by more than 30%.Both ablated settings still improve defense over No Defense.Source: §5.6 · p. 9, Table 4 · p. 9

6 Conclusion

The conclusion presents FATH as an authentication-based test-time defense against indirect prompt injection. The authors report state-of-the-art performance compared with existing test-time methods and describe it as an efficient way for developers to secure LLM-integrated applications.

6 Conclusion

  • FATH is an authentication-based test-time defense method for indirect prompt injection attacks.Source: §6 · p. 9
  • The authors report that FATH achieves state-of-the-art defense performance compared with existing test-time methods.Source: §6 · p. 9
  • The conclusion describes FATH as an efficient way for developers to secure LLM-integrated applications.Source: §6 · p. 9

Limitations

FATH requires substantial manual prompt design for each application and relies on LLM instruction-following ability. The evaluated benchmarks also lack real tool-use scenarios.

Limitations

  • FATH requires substantial manual defense-prompt design for each application, with prompt templates differing between OpenPromptInjection+ and InjecAgent.The authors identify automating defense-prompt design as future work.Source: Limitations · p. 10
  • FATH may be less effective on LLMs with comparatively weaker instruction-following abilities, such as Alpaca.The authors report efficacy using Meta-Llama-3-8B-Instruct.Source: Limitations · p. 10
  • Because existing benchmarks lack real tool usage, experiments simulate tool execution by directly providing external text information.The authors identify automating adaptive attacks and defense prompts as future work.Source: Limitations · p. 10

A.1 OpenPromptInjection Benchmark

The appendix shows FATH defense-prompt examples for OpenPromptInjection and InjecAgent, and identifies related prompt components.

A.1 OpenPromptInjection Benchmark

  • The OpenPromptInjection example pairs text summarization as the user task with sentiment classification as the injection task.The example appears in Figure 3.Source: §A.1 · p. 12

A.2 InjecAgent Benchmark

  • The InjecAgent appendix points to a defense-prompt example and separate figures for tool specifications and attack defense.The example is in Figure 4; the tool specifications and attack defense are in Figures 5 and 6, respectively.Source: §A.2 · p. 12

B Task Examples

The appendix describes benchmark task examples, an InjecAgent direct-harm example, attack templates, and baseline defense prompts.

B.1 OpenPromptInjection+

  • OpenPromptInjection+ examples cover a Stanford Alpaca target task and URL injection, question answering, and classification injection tasks.Each example contains a user instruction, external text information, and a ground-truth answer.Source: §B.1 · p. 12

B.2 InjecAgent

  • The InjecAgent direct-harm example attempts to grant Amy permanent August Smart Lock access instead of returning Amazon product reviews.The attack invokes AugustSmartLockGrantGuestAccess rather than AmazonGetProductDetails.Source: §B.2 · p. 12

C Attack Template for Threat Modeling 1

  • Threat Modeling 1 templates cover Naive Attack, Escape Characters, Context Ignoring, Fake Completion, and Combined Attack.The appendix also includes the Combined Attack template for InjecAgent.Source: C Attack Template for Threat Modeling 1 · p. 12

D.1 Baseline Defense Methods

  • The baseline prompts describe Instructional Prevention, Sandwich Prevention, Text Instruction Isolation, and In-context Learning defenses.The isolation baseline uses random strings as delimiters; the appendix includes the corresponding templates.Source: §D.1 · p. 12

D.2 Ablation Study

The appendix identifies two ablation settings through their defense-prompt templates.

D.2 Ablation Study

  • The ablation templates remove authentication tags in one setting and the security policy in another.The templates are shown in Figures 7 and 8, respectively.Source: §D.2 · p. 12

E.1 FATH and Baseline Defense Methods

The appendix presents adaptive-attack prompt templates for FATH and baseline defenses, plus templates for two ablation settings. It also frames the attack methods as red-teaming material and points developers to FATH as a mitigation strategy.

E.1 FATH and Baseline Defense Methods

  • Table 8 presents adaptive-attack prompt templates for FATH and baseline defense methods.The templates are in subsection E.1.Source: §E.1 · p. 12

E.2 Ablation Study

  • Table 9 presents adaptive-attack prompt templates for the ablation settings without Authentication Tags and without Security Policy.The templates are in subsection E.2.Source: §E.2 · p. 12

F Potential Risks

  • The paper provides attack methods for red-teaming purposes and hopes developers can use FATH to mitigate the described potential attacks.Source: F Potential Risks · pp. 12–13

G Dataset Details

The dataset details list sources and licenses for OpenPromptInjection+ and InjecAgent. They also provide examples and templates for benchmark tasks, defenses, and adaptive attacks.

G Dataset Details

  • OpenPromptInjection+ uses Stanford Alpaca under Apache-2.0, while InjecAgent data are collected and created under MIT; the listed injection-task sources carry CC BY 4.0 or MIT licenses.The passage names the fake Python package for URL data, Zverev et al. (2024) for QA instructions, and Liu et al. (2023b) for classification instructions.Source: G Dataset Details · p. 13
  • Table 5 provides examples for each task used in the OpenPromptInjection+ benchmark.Source: Table 5 · p. 14
  • Table 6 collects attack prompt templates for Threat Modeling 1, while Table 7 presents baseline defense prompt templates with replaceable contents highlighted.Source: Table 6 · p. 14, Table 7 · p. 14
  • The FATH defense prompt sets an instruction and external text, asks the model to detect additional instructions, and places the requested answer in an Authorized Section.For text without additional instructions, the example uses “SAFE TEXT INFORMATION” before the answer.Source: G Dataset Details · p. 19
  • The additional-instructions example marks the content as unauthorized, answers the clean instruction, and separates clean and injected answers into labeled sections.Source: G Dataset Details · p. 19
  • Tables 8 and 9 provide adaptive-attack templates for FATH and baseline defenses, and for ablations without Authentication Tags or Security Policy, respectively.The FATH template also includes attacker-generated replacement tags; Figures 7 and 8 show ablation templates without the tags and without the security policy.Source: Figure 7 · p. 19, Figure 8 · p. 20, Table 8 · p. 21

Related papers

Loading 2410.21492v2…