PaperlayerView exact revision on arXiv

Source-linked AI summary

SecAlign: Defending Against Prompt Injection with Preference Optimization

Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, Chuan Guo

arXiv:2410.05451v3cs.CRcs.LG

TL;DR

LLM-integrated systems can be manipulated when external data contains instructions that override the system’s intended task. SecAlign applies preference optimization to make models favor legitimate responses over injection-following responses, achieving low attack success rates while preserving utility in the reported evaluations.

  • Problem

    Prompt injection can make LLMs follow malicious instructions embedded in external data, while existing defenses are brittle against unseen attacks.

  • Method

    SecAlign constructs prompt-injected preference examples pairing secure responses to benign instructions with insecure responses to injections, then applies preference optimization.

  • Results

    SecAlign achieves mainly <10% attack success rates against stronger optimization-based attacks and maintains utility comparable to the non-preference-optimized model.

  • Takeaways & Limitations

    SecAlign provides a simple, utility-preserving defense with strong security against tested unseen and optimization-based prompt injection attacks.

  • Takeaways & Limitations

    SecAlign requires clearly separated instruction and data parts, cannot guarantee 100% security, and may be evaded by future attacks such as multi-turn injections.

Abstract

from arXiv · show

Large language models (LLMs) are becoming increasingly prevalent in modern software systems, interfacing between the user and the Internet to assist with tasks that require advanced language understanding. To accomplish these tasks, the LLM often uses external data sources such as user documents, web retrieval, results from API calls, etc. This opens up new avenues for attackers to manipulate the LLM via prompt injection. Adversarial prompts can be injected into external data sources to override the system's intended instruction and instead execute a malicious instruction. To mitigate this vulnerability, we propose a new defense called SecAlign based on the technique of preference optimization. Our defense first constructs a preference dataset with prompt-injected inputs, secure outputs (ones that respond to the legitimate instruction), and insecure outputs (ones that respond to the injection). We then perform preference optimization on this dataset to teach the LLM to prefer the secure output over the insecure one. This provides the first known method that reduces the success rates of various prompt injections to <10%, even against attacks much more sophisticated than ones seen during training. This indicates our defense generalizes well against unknown and yet-to-come attacks. Also, SecAlign models are still practical with similar utility to the one before defensive training in our evaluations. Our code is at https://github.com/facebookresearch/SecAlign

1 Introduction

LLM-integrated software systems are vulnerable to prompt injections that make models follow malicious instructions embedded in external data. Existing defenses separate instructions from data through prompting or fine-tuning, but fine-tuning defenses can fail against unseen attacks.

  • LLMs increasingly operate with user data, internet content, and external APIs in automated software systems.
  • Prompt injection attacks override system instructions by embedding malicious prompts in external inputs such as documents, retrieved data, or API results.The vulnerability is identified as a major security challenge for deploying LLM-integrated applications.
  • Existing defenses use prompting or fine-tuning to separate trusted instructions from untrusted data.Fine-tuning defenses maximize the likelihood of the benign response so the injected instruction is ignored.
  • Over 50% attack success rate was observed for StruQ under an attack that optimizes the injection.This example motivates concern about defense generalization to attacks unseen during fine-tuning.

A Secure High-Functioning SecAlign Model

SecAlign frames prompt-injection defense as preference optimization: the model learns to prefer responses that follow the legitimate instruction over responses that follow the injection. It reduces the strongest tested attack’s success rate to 8% while preserving utility, including across domains.

  • Results: 8% is the attack success rate reported for SecAlign against the strongest tested prompt injection.The figure describes this result for Llama3-8B-Instruct.
  • Motivation: Existing fine-tuning defenses may be fragile because they train desirable behavior without showing the model what an undesirable response looks like.SecAlign explicitly steers the model away from responses that follow the injected instruction.
  • Method: SecAlign formulates prompt-injection defense as preference optimization, teaching the LLM to prefer legitimate-instruction responses over injection-following responses.The method uses prompt-injected inputs paired with desirable and undesirable responses.
  • Novelty: Security against prompt injection is presented as a preference that can be optimized without human labor because the security policy is well-defined.The paper connects this formulation to preference optimization for aligning models with human values.
  • Results: SecAlign maintains the same utility as the non-preference-optimized counterpart whether its preference dataset matches or differs from the instruction-tuning domain.The evaluation covers five models and optimization-free and optimization-based attacks.
  • Follow-up: An improved SecAlign was used to build a 70B open-source commercial-grade LLM with built-in prompt-injection defense.The paper reports greater robustness than existing industry solutions, especially in agentic settings.

2 Preliminaries

The preliminaries define prompt injection as manipulation of an LLM into following instructions hidden in data rather than the system’s benign instruction. They describe the threat model, security objective, and manual and optimization-based attacks used for evaluation.

  • Prompt injection: Prompt injection is a test-time attack that hides a malicious instruction in external data and exploits the LLM’s instruction-following behavior.The attacker seeks execution of the injected instruction instead of the benign instruction supplied by the system designer.
  • Threat model: The attacker may inject an arbitrarily long instruction, knows the benign instruction and prompt format, and has white-box access to the target LLM.The benign instruction and prompt format cannot be modified by the attacker.
  • Objectives: A successful attack makes the LLM follow the injected instruction, whereas the defender aims to make it ignore injections and answer the benign instruction.The paper considers prevention-based defenses that require the LLM to answer the benign instruction.
  • Why it matters: Prompt injection can cause sensitive-information leakage and privacy breaches, threatening deployment of LLM-integrated applications if left unchecked.The paper highlights systems using web search, local documents, and cloud documents as vulnerable settings.
  • Attack classes: The paper evaluates manually constructed optimization-free attacks, including Ignore and Completion attacks, alongside more sophisticated optimization-based attacks.Completion attacks manipulate the benign response completion, while Ignore attacks explicitly request that previous instructions be disregarded.
  • Attack classes: GCG is adapted from jailbreaking to prompt injection by appending an optimized suffix to an input containing a benign instruction, data, and an injected prompt.The paper also adopts AdvPrompter and NeuralExec as optimization-based prompt-injection attacks.

3 Methodology

SecAlign formulates prompt-injection defense as preference optimization: it trains LLMs to favor responses to benign instructions over responses to injected instructions. Its preference dataset and objective explicitly discourage undesirable outputs, improving robustness while preserving utility in the reported evaluations.

  • Revisiting Prompt Injection Defenses: Existing defenses typically maximize the likelihood of the desirable response, but this objective does not directly reduce the likelihood of injected responses in LLMs.Because LLMs have V^L possible outputs, increasing one response’s probability has limited influence on many other sentences, including undesirable responses.
  • Formulating Prompt Injection Defense as Preference Optimization: DPO maximizes the log-likelihood margin between desirable and undesirable outputs while limiting deviation from the SFT reference model.The reference-model term constrains excessive deviation from the reference policy.
  • Evaluation: SecAlign lowers the average log probability of undesirable outputs to as low as -300, compared with -140 for StruQ, without influencing desirable outputs.The larger desirable-versus-undesirable margin indicates more effective adversarial training against prompt injections in the Llama-7B experiments.
  • Formulating Prompt Injection Defense as Preference Optimization: SecAlign treats prompt-injection defense as preference optimization between a desirable response y_w to the instruction and an undesirable response y_l to the injection.The method constructs prompt-injected inputs paired with both outputs and applies preference optimization to favor y_w over y_l.
  • Implementing SecAlign: Preference Dataset: SecAlign constructs its preference dataset by injecting another instruction into data, using the benign instruction’s response as y_w and the injected instruction’s response as y_l.The injected instruction is sampled from another instruction-tuning example, avoiding manually written injections; most examples use Straightforward attacks and 10% use Completion attacks.
  • Implementing SecAlign: Preference Dataset: The security policy enables preference-data generation without human labor because it is defined as responding to the benign instruction rather than the injection.The paper contrasts this with human-preference alignment, where safety policies cannot be rigorously written and require extensive feedback.

4 Experiments

Across utility, security, generalization, and ablation evaluations, SecAlign preserves model usefulness while substantially improving resistance to prompt injections, including unseen optimization-based attacks.

  • Utility and security: SecAlign does not decrease AlpacaEval2 WinRate on the evaluated Mistral-7B-Instruct and Llama3-8B-Instruct models, whose undefended counterparts exceed 70%.StruQ preserves utility on Mistral-7B-Instruct but reduces Llama3-8B-Instruct utility by around 4.5%.
  • Utility and security: 1% and 8% ASRs are achieved against the strongest optimization-based injections on the two evaluated Instruct models, versus StruQ’s 27% and 45%.The injections were unseen and completely different from those used in training.
  • Utility and security: 0% ASR is achieved against the Ignore attack across all five open-weight models, while instruction hierarchy records 1% ASR under the same optimization-free attack.SecAlign is evaluated against multiple optimization-free and optimization-based attacks.
  • Generalization: SecAlign achieves 0 ASR on the out-of-distribution InjecAgent API-calling benchmark, including long API-retrieved inputs, although that benchmark provides no utility evaluation.This result supports security generalization across a significantly different data domain.
  • Utility generalization: SecAlign improves handling of imperative data as data: its WinRate is 65.5%, and manual inspection finds 52% handled as data versus 16% for the undefended model.The evaluation compares Meta-Llama-3-8B-Instruct with and without SecAlign on 804 samples.
  • Ablation studies: DPO offers the reported balance between efficiency and performance: KTO has the best isolated security result but significantly higher runtime, while ORPO has doubled ASR.All three preference-optimization methods show similar utility performance.

5 Related Work

The paper situates prompt injection among broader LLM security and alignment research, distinguishing it from jailbreaking and related traditional injection threats. Existing work includes prompting- and fine-tuning-based defenses, while alignment methods such as DPO provide relevant methodological context.

  • Prompt injection attacks: Prompt injection manipulates LLM functionality by adding malicious instructions to trusted-system inputs, including externally retrieved content.The paper distinguishes direct injections, supplied by users, from indirect injections originating in external content such as web pages.
  • LLM alignment: Existing prompt-injection defenses use prompting or fine-tuning to separate trusted instructions from untrusted data and favor desirable responses.The paper frames fine-tuning defenses as related to the broader alignment objective of controlling model outputs through preferences.
  • Other threats to LLMs: Prompt injection differs from jailbreaking because the system instruction remains benign while the attacker embeds an instruction in data.Jailbreaking instead supplies a malicious instruction without accompanying data and seeks harmful or inappropriate outputs.
  • LLM alignment: DPO is presented as an offline alignment paradigm that trains language models from fixed datasets of human judgments.This alignment framework provides context for SecAlign’s use of preference optimization without requiring online reinforcement learning.

6 Conclusion and Discussions

SecAlign is presented as a fine-tuning defense that applies preference optimization to prompt-injection security, with reported utility preservation and strong security against tested attacks. The discussion also identifies deployment boundaries, including explicit instruction-data separation, untested attack settings, and the need for complementary defenses.

  • Conclusion: SecAlign is a state-of-the-art fine-tuning defense that connects LLM security with alignment through preference optimization.The authors characterize it as simple, utility-preserving, and robust to unseen attacks within the evaluated scope.
  • Limitations: SecAlign’s current scope requires explicit separation between instruction and data, such as delimiters, and does not guarantee complete security.The authors note possible evasion by future attacks, including prompt injections in multi-turn web-agent conversations.
  • Limitations: The method is most effective when injections occur at the end of data, although it generalizes strongly to other injection positions.Training with injections at varied positions is proposed as a possible strategy for improving security generation.
  • Limitations: SecAlign does not currently defend against threats outside prompt injection, including jailbreaking and data-extraction attacks.The limitation is specific to the method’s current defense target rather than a claim about LLM security generally.
  • Limitations: The authors do not regard SecAlign as a standalone solution and suggest combining it with detection, input reformatting, output manipulation, and system-level defenses.This multi-tiered approach is proposed for stronger security in LLM-integrated applications.
  • Limitations: Optimization-based prompt injections are currently too costly to generate inside SecAlign training, motivating future work on more efficient prompt optimization.The authors estimate that even a toy Alpaca dataset would require thousands of GPU hours with GCG in an inner training loop.
  • Real-world systems: Real-world applications may use more complicated prompt templates than the delimiter-separated setting studied in the paper.Retrieval augmentation is given as an example where retrieved text is appended to the model context and may contain injections.
  • Multimodal security: SecAlign has so far been evaluated on text-only LLMs, leaving protection for image- and speech-based prompt injections as an extension challenge.The authors suggest that multimodal instruction tuning may support extending the approach to additional input modalities.

Appendix

The appendix provides additional breakdowns and robustness checks for SecAlign’s security evaluation. These include attack-specific comparisons, alternative injected words, and a broader metric that also counts false positives.

  • Attack breakdowns: Table 6 breaks down the attack results, identifying Ignore as the weakest optimization-free attack and GCG as the strongest tested attack on the authors’ SFTed models.AdvPrompter is stronger on industry-tuned instruction models according to the appendix summary.
  • Defense comparison: Table 7 reports the breakdown underlying the comparison in Table 2, where SecAlign significantly outperforms existing prompting-based defenses.
  • Robustness checks: Across alternative injections, “Hello World” lowers optimization-free attack success rates while “Serendipity” raises them, and SecAlign remains at 0% ASR.The appendix also reports a broader criterion that recalls all occurrences of the injected word but includes false positives.
Loading 2410.05451v3…