PaperlayerView exact revision on arXiv

Source-linked AI summary

DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks

Yupei Liu, Yuqi Jia, Jinyuan Jia, Dawn Song, Neil Zhenqiang Gong

arXiv:2504.11358v4cs.CRcs.AI

TL;DR

Prompt injection attacks contaminate untrusted data to redirect LLM-integrated applications, while existing detectors have limited effectiveness, especially against adaptive attacks. DataSentinel fine-tunes a detection LLM through a game-theoretic minimax procedure that alternates attack optimization and detector updates. Evaluations across multiple tasks, attacks, datasets, and LLMs show strong detection of existing and adaptive attacks within its supported scope.

  • Problem

    Existing prompt-injection detectors have limited effectiveness against state-of-the-art and adaptive attacks, despite the need to identify contaminated target data while maintaining low false alarms.

  • Method

    DataSentinel fine-tunes a detection LLM by formulating adaptive attack detection as a minimax problem and approximately solving it by alternating inner maximization and outer minimization.

  • Results

    DataSentinel is highly effective across 7 target tasks, 9 injected tasks, 6 LLMs, and 9 existing attacks, with FPR close to 0 and FNR at most 0.07 in the reported experiments.

  • Takeaways & Limitations

    DataSentinel effectively detects existing and adaptive prompt injection attacks when contaminated target data includes injected instructions that induce an injected task differing from the target task.

  • Takeaways & Limitations

    DataSentinel is less effective when injected and target tasks are the same, because adaptive adversarial examples may contain injected data without injected instructions.

Abstract

from arXiv · show

LLM-integrated applications and agents are vulnerable to prompt injection attacks, where an attacker injects prompts into their inputs to induce attacker-desired outputs. A detection method aims to determine whether a given input is contaminated by an injected prompt. However, existing detection methods have limited effectiveness against state-of-the-art attacks, let alone adaptive ones. In this work, we propose DataSentinel, a game-theoretic method to detect prompt injection attacks. Specifically, DataSentinel fine-tunes an LLM to detect inputs contaminated with injected prompts that are strategically adapted to evade detection. We formulate this as a minimax optimization problem, with the objective of fine-tuning the LLM to detect strong adaptive attacks. Furthermore, we propose a gradient-based method to solve the minimax optimization problem by alternating between the inner max and outer min problems. Our evaluation results on multiple benchmark datasets and LLMs show that DataSentinel effectively detects both existing and adaptive prompt injection attacks.

1. Introduction

LLM-integrated applications process external data with backend LLMs, creating vulnerability to prompt injection attacks. DataSentinel addresses weaknesses in existing detection by fine-tuning a detection LLM against adaptive attacks through game-theoretic optimization, and performs strongly across broad evaluations.

  • Application setting: LLM-integrated applications concatenate target instructions and externally sourced target data before querying a backend LLM.Examples include webpage summarization, where webpages relevant to a search query serve as target data.
  • Prompt injection threat: Injected prompts can redirect an application from its target task to an attacker-chosen injected task.Attackers contaminate untrusted target data by embedding injected instructions and data.
  • Existing detection: Known-answer detection uses a standard detection LLM and a secret-key instruction, but retains high false positive and false negative rates against strong adaptive attacks.Its limitations include lacking detection-specific training and not accounting for adaptive attacks by design.
  • DataSentinel: DataSentinel fine-tunes the detection LLM to become more vulnerable to injected prompts, turning that vulnerability into a detection mechanism.It also incorporates adaptive attacks that seek to evade detection while misleading the backend LLM.
  • Optimization: DataSentinel approximately solves a minimax objective by alternating inner maximization of injected prompts with outer minimization updating the detection LLM.The alternating process repeats for a predefined number of rounds.
  • Evaluation: Across 7 target tasks, 9 injected tasks, 6 LLMs, and 9 existing attacks, DataSentinel achieves FPR close to 0 and FNR close to 0 for several attacks, at most 0.07 for others.It significantly outperforms 6 baseline detection methods; known-answer detection reaches FPR up to 0.1 and FNR up to 0.21 in specified settings.

2. Related Work

Related work frames prompt injection as redirecting an LLM-integrated application to an injected task, while distinguishing it from adversarial examples that preserve the target task. Existing defenses include prevention and detection, but detection remains limited.

  • Application model: An LLM-integrated application concatenates a target instruction and target data, then queries a backend LLM to produce the target output.The backend LLM generates its response autoregressively from this concatenated prompt.
  • Prompt injection attacks: Prompt injection attacks contaminate target data with an injected prompt so the backend LLM performs an attacker-chosen injected task instead of the target task.Heuristic attacks manually add separators, while optimization-based attacks optimize separators, prompts, or entire contaminated inputs.
  • Adversarial examples: Traditional adversarial examples keep the target task unchanged while seeking an incorrect output, unlike prompt injection attacks that often change the task itself.Adaptive adversarial examples are notoriously difficult to detect.
  • Defenses: Defenses against prompt injection attacks include prevention methods that alter data or backend behavior and detection methods that identify contaminated target data.Some prevention approaches require task-specific fine-tuning or may remain vulnerable to new attacks.
  • Known-answer detection: Known-answer detection asks a detection LLM to produce a defender-held secret key after receiving a detection instruction concatenated with target data.Failure to output the key suggests that the target data contains an injected prompt, but existing detection methods still have limited effectiveness.
  • DataSentinel: DataSentinel differs by fine-tuning the detection LLM rather than using an otherwise standard detection model.Its design targets the limitations of existing detection methods and adaptive attacks.

3. Problem Formulation

The problem formulation defines prompt-injection detection, its accuracy goals, and the attacker and defender settings considered by the paper.

  • 3.1. Threat Model: Attackers aim to contaminate target data so an application performs an attacker-chosen injected task instead of its target task.The threat model considers attacker-desired outputs that accomplish the injected task.
  • 3.1. Threat Model: The attacker is assumed to have strong background knowledge, including access to the LLM-integrated application’s details.The paper further assumes the attacker knows the detector’s details.
  • 3.2. Detecting Prompt Injection Attacks: The defender develops a detector either for a specific backend LLM or for applications using different backend LLMs.The defender may be an application developer or a third-party detector provider.
  • 3.2. Detecting Prompt Injection Attacks: Detection determines whether target data is contaminated with an injected prompt.The detector takes target data x as input and outputs “contaminated” or “clean”.
  • 3.2. Detecting Prompt Injection Attacks: The detector should achieve small false positive and false negative rates.FPR measures clean data falsely labeled contaminated, while FNR measures contaminated data falsely labeled clean.
  • 3.2. Detecting Prompt Injection Attacks: A large false positive rate raises many false alarms and may cause applications to abandon the detector.The formulation therefore emphasizes maintaining a small FPR while detecting as much contaminated data as possible.

4. Our DataSentinel

DataSentinel detects contaminated target data by checking whether a defender-known secret key appears in a detection LLM’s output. It fine-tunes that detector against strategically optimized attacks through an alternating minimax procedure.

  • Detection mechanism: DataSentinel classifies target data as contaminated when the detection LLM omits a defender-known secret key from its output.The detector receives a detection instruction concatenated with the target data.
  • Detector fine-tuning: The method fine-tunes the detection LLM to omit the secret key for contaminated data and include it for clean data, reducing false-negative and false-positive errors.The fine-tuning dataset contains both contaminated and clean target data.
  • Game-theoretic formulation: DataSentinel models adaptive attacks and detector fine-tuning as an inner-max, outer-min game, then solves the minimax problem with gradient-based alternating updates.Each round first optimizes contaminated target data against the current detector, then updates the detector using the optimized data and clean tasks.
  • Detection mechanism: A detection instruction can use any defender-known ground-truth answer unavailable to attackers, such as repeating a secret key while ignoring following text.The experiments use a template that asks the model to repeat the secret key once while ignoring the following text.
  • Adaptive attack: The adaptive attack jointly seeks to evade detection and induce the backend LLM to perform an injected task.Its objective combines a secret-key loss for the detector with a task-output loss for the backend LLM, balanced by a hyper-parameter.
  • Optimization procedure: The defender constructs target and injected task sets from a task collection and repeatedly solves the inner problem for contaminated data before solving the outer detector-update problem.The task collection can be formed from standard benchmark datasets; the injected tasks simulate hypothetical strong adaptive attacks.
  • Scope and caveat: A constraint on contaminated data may weaken the adaptive attack, yet experiments report effective detection of existing and strong adaptive prompt injection attacks.The reported rationale is that constrained contaminated data remains adaptive to the detector and contains injected prompts.

5. Evaluation

DataSentinel is evaluated across diverse tasks, attacks, LLMs, and datasets, with results showing low error rates, strong performance against baselines and adaptive attacks, and manageable overhead. Ablations examine data size, optimization parameters, and iteration counts, while a third-party setting tests detection without backend-LLM access.

  • Main Results: FPRs are close to 0, while FNRs are no larger than 0.07 across existing heuristic-based and optimization-based attacks.Heuristic-based attacks have FNRs of 0 in the reported comparison, whereas optimization-based attacks can have non-zero FNRs up to 0.07.
  • Main Results: DataSentinel significantly outperforms baseline detectors on both FPR and FNR, with larger advantages over KAD against adaptive attacks.The comparison attributes the advantage to fine-tuning the detection LLM to be more vulnerable to prompt injection attacks.
  • Efficiency: Detection requires one detection-LLM query averaging 1.6 seconds, versus 15.3 seconds for a backend-LLM query, making detection overhead around 10%.A fine-tuned LLaMA3.2-1B-Instruct detector achieves average FPR 0.00 and FNR 0.01 with 0.7-second query time.
  • Third-Party Setting: DataSentinel remains effective against NeuralExec attacks generated for different backend LLMs, supporting use by third-party providers without backend-LLM access.The reported FNRs are averaged over 7×7 target-injected task pairs.
  • Adaptive Attacks: Under adaptive attacks, DataSentinel’s FNR is at most 0.06 for all target tasks except sentiment analysis, where matching target and injected tasks yields an FNR of 0.87.The reported explanation is that same-type prompt injection attacks reduce to traditional adversarial examples, which are difficult to detect.

6. Discussion and Limitations

The discussion identifies both a scope limitation of DataSentinel and a motivation for detection despite imperfect backend robustness. It also shows why detection-LLM behavior does not reliably predict backend-LLM behavior.

  • If the backend LLM were perfectly robust against prompt injection, detection would not be necessary.
  • KAD evasion does not reliably imply that the backend LLM will ignore the injected instruction, because detection and target-task contexts differ.
  • Some contaminated samples that evade detection still cause the backend LLM to complete the injected task successfully.
  • Even unsuccessful injected tasks can produce untargeted attacks when the backend LLM also fails to complete the target task correctly.
  • DataSentinel is less effective when injected and target tasks are of the same type, where adaptive adversarial examples may contain injected data without injected instructions.

7. Conclusion and Future Work

The paper concludes that DataSentinel uses minimax fine-tuning to account for strong adaptive attacks and performs effectively when contaminated data includes injected instructions. Future work targets stronger attacks and multimodal models.

  • DataSentinel formulates detector fine-tuning and strong adaptive attacks as a minimax optimization problem.
  • Evaluation shows the detector is highly effective against existing and adaptive prompt injection attacks when contaminated target data includes injected instructions.
  • Future work includes stronger adaptive attacks and extending DataSentinel to multimodal models.
  • The evaluation uses seven natural language processing tasks, each serving as either a target task or an injected task.

Appendix B. Details of Heuristic-based Attacks

The heuristic-based attacks represent contaminated data by concatenating target data, an attack-specific separator, a special element, and injected data.

  • Heuristic-based contaminated data is represented as x_c = x_t||z||s_e||x_e.
  • The separator z distinguishes the heuristic-based prompt injection attacks.
  • For Naive Attack, the separator is an empty text string.

Appendix C. Details of DataSentinel (Min)

The appendix details the Min variant’s setup, examples, and evaluation materials, including task-level false-positive rates and tables covering attack-specific detection results and attack effectiveness metrics.

  • DataSentinel (Min) uses Mistral-7B as the detection LLM, β = 1, 3,000 iterations, batch size 2, and 500 Gigaword training samples.
  • The FPRs of DataSentinel (Min) across seven natural language tasks are 0.01, 0.00, 0.00, 0.04, 0.00, 0.03, and 0.01, respectively.
  • Table 7 lists separators for evaluation attacks and adaptive attacks generated during fine-tuning, with an example of optimized contaminated target data.
  • The detection LLM can output a known secret key when given a detection instruction and data sample.
  • ASV measures attack effectiveness, with larger ASV indicating a more effective attack; optimization-based adaptive attacks may have method-specific ASVs.

Appendix D. Meta-Review

The meta-review situates DataSentinel as a finetuned extension of Known Answer Detection for detecting prompt injection attacks.

  • The meta-review describes the paper as proposing DataSentinel, which uses a finetuned LLM to detect prompt injection attacks.
  • DataSentinel builds on Known Answer Detection, which uses a secondary LLM to check whether the primary LLM followed a hidden instruction.
  • The proposed finetuning is intended to reduce the detection scheme’s error rate.

D.2. Scientific Contributions

The paper’s scientific contribution is presented as a valuable advance in an established field of prompt-injection defense.

  • The meta-review characterizes the paper as a valuable step forward in an established field.
  • Its contribution is framed as advancing existing work rather than introducing an unrelated research direction.
  • The contribution is identified within the established field addressed by the paper.

D.3. Reasons for Acceptance

The stated reason for acceptance is that DataSentinel introduces a minimax objective for finetuning the detection LLM and substantially improves defense effectiveness.

  • DataSentinel proposes a minimax optimization objective for finetuning the detection LLM.
  • The objective is applied within the Known Answer Detection scheme.
  • The meta-review says this approach drastically improves the effectiveness of the defense.

D.4. Noteworthy Concerns

A concern is that DataSentinel’s defense may become less effective as LLMs improve at following instructions, because adaptive attacks could exploit that capability.

  • The defense may work less well as LLMs become better at following instructions.
  • Improved instruction following might make adaptive attacks easier to construct.
  • Such attacks could make the LLM return the known answer while following the prompt injection.
Loading 2504.11358v4…