Source-linked AI summary
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, David Wagner
TL;DR
Prompt injection lets malicious input hijack an LLM’s instruction-following, threatening LLM-integrated applications. Jatmo uses teacher-generated task data to fine-tune non-instruction-tuned base models for specific tasks. Across seven tasks, Jatmo retained similar output quality while reducing reported attack success from 87% against GPT-3.5-Turbo to 0.5% against its models.
Problem
Prompt injection can hijack LLMs’ instruction-following and change their responses, posing a threat to LLM-integrated applications that process untrusted data.
Source: §1 · pp. 1–2Method
Jatmo uses an instruction-tuned teacher to generate task-specific training data, then fine-tunes a non-instruction-tuned base model for the task.
Source: §1 · p. 3, §4 · p. 9, §4.1 · p. 10Results
Across seven tasks, the best attacks succeeded at a 0.5% rate against Jatmo models versus 87% on average against GPT-3.5-Turbo, with negligible response-quality loss.
Source: §1 · p. 3Takeaways & Limitations
Jatmo provides task-specific models that are resilient to prompt injections while maintaining similar output quality to standard LLMs on their tasks.
Source: §5 · p. 11, §5.2 · p. 13Single-task models sacrifice versatility, and how to build a general-purpose model secure against prompt injection remains open.
Source: §6 · p. 16
Abstract
from arXiv · showhide
Large Language Models (LLMs) are attracting significant research attention due to their instruction-following abilities, allowing users and developers to leverage LLMs for a variety of tasks. However, LLMs are vulnerable to prompt-injection attacks: a class of attacks that hijack the model's instruction-following abilities, changing responses to prompts to undesired, possibly malicious ones. In this work, we introduce Jatmo, a method for generating task-specific models resilient to prompt-injection attacks. Jatmo leverages the fact that LLMs can only follow instructions once they have undergone instruction tuning. It harnesses a teacher instruction-tuned model to generate a task-specific dataset, which is then used to fine-tune a base model (i.e., a non-instruction-tuned model). Jatmo only needs a task prompt and a dataset of inputs for the task: it uses the teacher model to generate outputs. For situations with no pre-existing datasets, Jatmo can use a single example, or in some cases none at all, to produce a fully synthetic dataset. Our experiments on seven tasks show that Jatmo models provide similar quality of outputs on their specific task as standard LLMs, while being resilient to prompt injections. The best attacks succeeded in less than 0.5% of cases against our models, versus 87% success rate against GPT-3.5-Turbo. We release Jatmo at https://github.com/wagner-group/prompt-injection-defense.
1 Introduction
Prompt injection threatens LLM-integrated applications because malicious data can take control of model responses. Jatmo fine-tunes task-specific models to resist these attacks, and its seven-task evaluation reports low attack success with little quality loss.
1 Introduction
- Prompt injection is a threat because malicious data can override an application’s prompt and control the LLM’s response.The paper describes prompt injection as a major risk when applications process data from untrusted sources.Source: §1 · pp. 1–2
- Jatmo replaces general-model invocation on prompt-plus-data with a single-task model fine-tuned from a non-instruction-tuned base model.The resulting model processes data for its task without being trained to follow instructions in its input.Source: §1 · pp. 2–3
- When no input dataset exists, Jatmo can generate synthetic task inputs and fine-tune a model from them.The approach can also use an existing input dataset and teacher-generated outputs, or gold-standard outputs when available.Source: §1 · p. 3
- 87% average attack success against GPT-3.5-Turbo falls to 0.5% against Jatmo in the reported experiments.Only two of 23,400 prompt-injected inputs succeeded against a Jatmo model.Source: §1 · p. 3
- Jatmo requires a separately trained model for each unique task prompt, with upfront fine-tuning costs that make one-off interactive prompts unsuitable.The paper focuses on API applications with a fixed trusted prompt and runtime-varying data, not web chat.Source: §1 · pp. 2–3
2 Background and Related Work
The background describes how LLMs are trained and used in applications, then surveys prompt injection and other attacks alongside proposed defenses.
2 Background and Related Work
- LLMs are pretrained for text completion, then may be fine-tuned for instructions, rules, or multi-turn conversation and further trained with human feedback.The passage cites [59] for fine-tuning and [6] [40] for reinforcement learning from human feedback.Source: §2 · p. 4
- LLM-integrated applications can use zero-shot prompts that combine developer-provided instructions with user inputs, sometimes separated by delimiters.The passage cites [21] for zero-shot prompting and for instruction templates.Source: §2 · p. 4
- Prompt injection is described as a malicious prompt that redirects an LLM from its intended task, while surveyed defenses include prompting, filtering, and LLM-based detection.The authors state that, to their knowledge, no existing defenses were effective; the cited approaches include [31], [3], and.Source: §2 · p. 4
- The discussion distinguishes prompt injection from jailbreak attacks targeting alignment [20] [10], data-extraction attacks eliciting training data or personally identifiable information, and task-specific attacks decreasing LLM performance [60] [52].Jailbreak attacks are cited as [15] [9] [55]. Listed defenses include paraphrasing or retokenization [19], perplexity detection [19], LLM-based detection [25], randomized smoothing [46], and in-context demonstration [55].Source: §2 · p. 4
3 Problem Statement
The problem statement defines the attack and threat model, illustrates prompt injection, distinguishes attack types, and explains why common defenses do not fit LLM applications. Jatmo draws on query parameterization to separate task instructions from potentially untrusted data.
3.1 Definition
- An attack succeeds when an attacker-controlled input makes the victim follow a malicious instruction instead of the legitimate one; merely failing the legitimate task does not count.The attacker controls input data but not the legitimate instruction, and the authors make no assumption about the attacker’s knowledge of the model or instruction.Source: §3.1 · pp. 4–5
3.2 Examples
- The examples show code comments redirecting a summarization model to emit a command and an injected customer review hijacking a review-summary task.In the review example, the injected text directs the model to print “HACKED” instead of summarizing reviews.Source: §3.2 · pp. 5–6
3.3 Types of Attacks
- The paper distinguishes goal hijacking from prompt leaking and evaluates goal hijacking, while stating that Jatmo also defends against prompt leaking.It discusses both direct attacks on chatbots and indirect attacks through external data sources such as retrieved documents or websites.Source: §3.3 · pp. 6–7
3.4 Pitfalls of Traditional Defenses
- Input sanitization is difficult because attacks can be obfuscated, delimiters do not reliably prevent instruction following, and removing suspected instructions can harm task performance.The text names language changes, encoding, paraphrasing, and payload splitting as possible obfuscation strategies.Source: §3.4 · p. 8
- Output verification is inadequate for many free-form tasks: syntactic checks miss malicious but valid text, while checking correctness may require human review or another vulnerable LLM.The passage contrasts easy JSON syntax checks with the harder task of verifying that an answer accurately reflects the data.Source: §3.4 · pp. 8–9
- Query parameterization separates control from data but suits rigid programmatic interfaces, whereas LLMs commonly mix them in natural-language input.Jatmo adapts the separation principle by creating a specialized model with a safe-by-design parameterized interface.Source: §3.4 · p. 9
4 Jatmo
Jatmo creates task-specific models by fine-tuning a non-instruction-tuned base model, using a teacher model to generate task outputs. When task inputs are unavailable, it can generate a synthetic dataset from the task prompt, optionally guided by example inputs.
4 Jatmo
- Jatmo’s rationale is that a task-fine-tuned base model that has not undergone instruction tuning should not follow malicious instructions.The method targets prompt-injection resilience by specializing the base model to a single task.Source: §4 · p. 9
- Jatmo collects task inputs, uses a teacher model with the task prompt to generate outputs, then fine-tunes the base model on those input-output pairs.Part of the dataset is reserved for quality and prompt-injection evaluations.Source: §4 · pp. 9–10
4.1 Synthetic Input Generation
- When no input or output examples are available, Jatmo can build a fully synthetic fine-tuning dataset using the task prompt and optionally example inputs.The ordinary procedure uses existing inputs when available and a teacher model to produce their outputs.Source: §4.1 · p. 10
- GPT-4 generates 10 seed inputs, expands them into a larger dataset, and normalizes formatting using either a real example or a generated template.Generating from seeds is intended to keep inputs structurally similar while maintaining diversity.Source: §4.1 · pp. 10–11
5 Results
Across seven tasks, Jatmo models retained teacher-like task quality while resisting prompt injections, including when trained on automatically generated data. The results examine evaluation methods, reduced training sets, and synthetic-data generation.
5.1 Experimental Methodology
- Seven-task evaluation fine-tuned davinci-002 on GPT-3.5-Turbo labels and tested quality plus attacks placed at three input positions.The attack set combined generic HackAPrompt injections with manually crafted task-specific attacks.Source: §5.1 · pp. 11–13
5.2 Main Results
- Across seven tasks, Jatmo quality stayed within 2% of GPT-3.5-Turbo while resisting almost all tested prompt injections.The results include models trained on real or synthetic data.Source: §5 · p. 11, §5.2 · p. 13
- Jatmo’s only fully successful tested attack hit review summarization at 2% when placed at input end; GPT-3.5-Turbo averaged 97% at input ends and 67% mid-input.Manual inspection of news summaries found four partially successful attacks among 3,900 tested injections; the authors note their success metric can underestimate attack effectiveness.Source: §5.2 · p. 13
5.4 Synthetic Dataset Generation
- 96% of GPT-3.5-Turbo’s quality across tasks came from synthetic-data training with one real example at T=0.7.Zero-shot inputs could be diverse but generic topics hurt performance; one real example made generated data more representative of real inputs.Source: Figure 5 · p. 14, §5.4 · pp. 14–15
5.3 Training with Less Data
- News summarization matched teacher quality with 100 examples, while product-review summarization did so with 10; all three plotted tasks did so by 400.The authors attribute variation partly to differences in task-data diversity and GPT-3.5 pretraining.Source: Figure 4 · p. 13, §5.3 · p. 14
5.4 Synthetic Dataset Generation
- With ten real examples, synthetic datasets for code summarization and translation trained on 800 samples gained 2% quality over one-shot models.The authors say synthetic examples are less diverse than original data and about twice as many are needed for similar performance.Source: §5.4 · pp. 15–16
6 Discussion
Jatmo specializes models for one task, leaving general-purpose prompt-injection defense unresolved. Its scope excludes jailbreaks and adversarial examples, and its evaluation may not cover every attack strategy.
6 Discussion
- Single-task models sacrifice versatility, and how to build a general-purpose model secure against prompt injection remains open.The authors consider the trade-off potentially acceptable for applications intended to perform a specific task.Source: §6 · p. 16
- Jatmo addresses prompt-injection attacks, not jailbreaks against alignment or adversarial examples.Source: §6 · p. 16
- Evaluation covered currently known prompt-injection strategies, but more sophisticated attacks may have been missed.The authors welcome further security evaluation.Source: §6 · p. 16
- Jatmo requires fine-tuning a non-instruction-tuned base model; the authors expect fine-tuning an instruction-tuned model would not prevent prompt injection.The paper notes that providers increasingly offer only instruction-, chat-, and alignment-tuned models.Source: §6 · p. 16
7 Summary
Jatmo creates task-specific LLMs by using an instruction-tuned model to bootstrap data for fine-tuning a separate base model. The paper reports similar task performance in most cases alongside a sharp reduction in prompt-injection success.
7 Summary
- Jatmo bootstraps a task dataset with an instruction-tuned model, then fine-tunes a different base model into a task-specific model.Source: §7 · p. 16
- Prompt-injection success falls from 87% to approximately 0% while task-specific models match standard-model performance in most cases.Source: §7 · p. 16
- The authors suggest Jatmo as a practical method for protecting LLM-integrated applications against prompt injection.Source: §7 · p. 16
A.1 Detailed task parameters.
The appendix gives examples of task prompts and corresponding injections that try to override them. Across the examples, attacks redirect summarization, classification, similarity scoring, and other tasks toward attacker-chosen outputs.
A.1 Detailed task parameters
- Code and news summarization injections tell the model to disregard the original task and print “HACKED” instead.Source: §A.1 · p. 21
- Other examples redirect a command-related task to output a shell command and a sentiment task to return the attacker’s requested label, “hack.”Source: §A.1 · p. 21
- A toxicity-detection injection instructs the model to ignore the comment and output “HACKED,” while a similarity attack prompts it to say “I UNDERSTAND” before scoring.Source: §A.1 · p. 22
A.2 Synthetic dataset generation prompts.
The appendix describes prompts for generating synthetic task inputs and preparing them for fine-tuning. It emphasizes realistic, varied examples and consistent formatting, including when no example input is available.
A.2 Synthetic dataset generation prompts
- When no example is provided, the prompt omits the example-dependent text; unique indices and random seeds help make requests distinct and promote input diversity.Source: §A.2 · p. 22
- Generated inputs should be unique, realistic, long-form, high-quality, and formatted like real task data, with positive and negative examples for classification tasks.Source: §A.2 · p. 23
- If inputs lack a format template, the process first generates a formatted example, then uses a second query to reformat the inputs consistently.Source: §A.2 · p. 23
- The template-conversion prompt preserves original content and separators, uses “N/A” for absent expected fields, and allows unmatched original fields to be omitted.Source: §A.2 · p. 24