Source-linked AI summary
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, Alex Beutel
TL;DR
LLMs often fail to distinguish trusted system instructions from untrusted user and third-party content, enabling attacks that override original instructions. The paper introduces an instruction hierarchy trained through synthetic data generation and context distillation. GPT-3.5 becomes more robust across seen and unseen attacks while generic capabilities remain comparable, though over-refusals and powerful adversarial attacks remain concerns.
Problem
LLMs often treat system prompts and untrusted user or third-party content as having the same priority, leaving them vulnerable to instruction-overriding attacks.
Method
The paper defines an instruction hierarchy and trains models with context synthesis for aligned instructions and context ignorance for misaligned instructions.
Results
Robustness improves across all main evaluations by up to 63% and on excluded attack criteria by up to 34%, while capability metrics remain comparable to the baseline.
Takeaways & Limitations
The learned hierarchy improves safety and controllability, including for unseen jailbreaks, password-extraction attacks, and tool-use prompt injections.
Takeaways & Limitations
Current models may remain vulnerable to powerful adversarial attacks, and the training currently handles conflicting instructions conservatively during browsing or tool use.
Abstract
from arXiv · showhide
Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities.
1 Introduction
LLM applications face attacks because models do not reliably distinguish trusted instructions from untrusted user and third-party content. The paper proposes an instruction hierarchy and training-data strategy to prioritize higher-privileged instructions while preserving benign behavior.
- Motivation: Agentic LLM applications can expose private data or perform unsafe actions when adversaries manipulate the model.Examples include email assistants that could exfiltrate users’ private emails.
- Problem: The paper identifies the lack of instruction privileges as a mechanism underlying jailbreaks, system prompt extraction, and direct or indirect prompt injections.These attacks can override application restrictions or exploit third-party tool content.
- Instruction hierarchy: The proposed hierarchy gives system messages precedence over user messages, and user messages precedence over third-party content.This ordering is intended to resolve conflicts between instructions at different privilege levels.
- Training approach: Training data uses synthetic compositional examples and context distillation to teach models to follow aligned lower-level instructions and ignore misaligned ones.Decomposed instructions are placed at different hierarchy levels, while misaligned examples use responses generated as if lower-level instructions were unseen.
- Results: 63% improved defense against system prompt extraction and over 30% increased jailbreak robustness demonstrate stronger safety, with some over-refusals but otherwise preserved generic capabilities.The evaluations include attacks unlike those seen during training; the authors report that generic capabilities remain otherwise unscathed.
2 Background: Attacks on LLMs
LLM applications process structured messages from application builders, users, models, and third-party tools. Attacks exploit conflicts or weaknesses across these inputs, including prompt injections, jailbreaks, and system-message extraction.
- Message types: Messages serve different purposes and use special tokens so LLMs can distinguish their types.The structured input includes system messages, user messages, model outputs, and tool outputs.
- Message types: System messages define general instructions, safety guidelines, constraints, and available tools, while user messages contain end-user inputs.Model outputs are generated responses, and tool outputs provide results from browsing, code execution, or third-party APIs.
- Prompt injections: Prompt injections insert instructions that subvert the system designer’s intent and can cause harms such as data exfiltration or application hijacking.They target applications built on models, especially when those applications access confidential data or act in the world.
- Prompt injections: Direct prompt injections come from end users, whereas indirect prompt injections arrive through third-party inputs such as browsing or tool-use results.The distinction concerns how the adversarial instruction enters the model’s input.
- Jailbreaks: Jailbreaks aim to escape trained safety behavior and can enable malicious outputs such as spam, misinformation, or pornographic content.Unlike many prompt injections, jailbreaks need not specifically conflict with previous model instructions.
- System-message extraction: System-message extraction attacks seek the full system message or specific secrets because system prompts may contain business logic, private information, or safety instructions.Knowing the system prompt can aid further attacks.
3 The Instruction Hierarchy
The instruction hierarchy teaches LLMs to defer to higher-privileged instructions during conflicts while conditionally following aligned lower-level instructions. Training data combines synthetic generation with context synthesis and context ignorance across several attack settings.
- 3 The Instruction Hierarchy: The proposed hierarchy makes LLMs defer to higher-privileged instructions when conflicts arise.The hierarchy places system messages above user messages and user messages above third-party content.
- 3.1 Overview of Ideal Model Behavior: Aligned lower-level instructions should be followed, whereas misaligned instructions should be ignored or refused when proceeding is impossible.This behavior is conditioned on alignment with higher-level constraints, rules, or goals.
- 3.2 Training Data Generation for Different Attacks: Training data generation uses synthetic data generation and context distillation to teach hierarchical behavior.The paper proposes separate approaches for aligned and misaligned instructions.
- 3.2 Training Data Generation for Different Attacks: Context synthesis decomposes compositional requests, places their pieces at different hierarchy levels, and trains models to reproduce the original response.This approach is used for aligned instructions in open-domain tasks.
- 3.2 Training Data Generation for Different Attacks: Context ignorance trains models to produce the answer they would give without lower-level instructions, or to refuse when the task cannot proceed.It is used for misaligned instructions, including adversarial user requests and some prompt-injection settings.
- 3.2 Training Data Generation for Different Attacks: Closed-domain tasks are trained only on misaligned instructions because the model should perform the developer-specified task regardless of inserted user instructions.Prompt injections are generated for each task, and ground-truth responses are obtained through context distillation.
- 3.2 Training Data Generation for Different Attacks: Indirect prompt injections are treated as misaligned during browsing or tool use and trained with context ignorance using automatically generated attacks in search results.The current hierarchy assumes instructions appearing during browsing or tool use should be ignored.
- 3.2 Training Data Generation for Different Attacks: System-message extraction training distinguishes refusals for explicit or secret requests from compliance with basic inquiries about available system capabilities.Training excludes attacks seeking private information or passwords to test generalization.
4 Main Results
The instruction hierarchy improves safety robustness across main and held-out evaluations while preserving generic capabilities, though some adversarial over-refusal tasks regress.
- Experiment Setup: GPT-3.5 models fine-tuned with instruction-hierarchy data achieved comparable capability metrics to the baseline on TriviaQA, LAMBADA, and HellaSwag.Both models used supervised fine-tuning and reinforcement learning from human feedback; the baseline lacked instruction-hierarchy examples.
- Main Results: Up to 63% higher robustness was observed across the main safety evaluations.The evaluation suite included in-domain attacks, generalization attacks, and over-refusal evaluations, with metrics framed so higher is better.
- Generalization Results: Up to 34% higher robustness generalized to evaluation criteria explicitly excluded from training, including jailbreaks, system-password extraction, and tool-use prompt injections.The authors interpret these results as evidence that the model learned to internalize the instruction hierarchy for unseen prompts.
- Qualitative Examples: Qualitative adversarial tests show the model treating user input as data rather than instructions in one case and correctly refusing in two others.One example contrasts baseline output “PLANETARY” with the instruction-hierarchy model’s refusal.
- Over-refusal Results: Over-refusal evaluations found near-baseline compliance on non-conflicting instructions, but regressions on System Message Probing Questions and Jailbreakchat with Allowed Prompts.These tasks contain benign or boundary-case inputs that resemble attacks; the authors do not expect noticeable degradation in typical real-world use.
5 Discussion & Related Work
The discussion positions the instruction hierarchy as a multi-level alternative to defenses that simply ignore user-provided instructions, complemented by broader system-level safeguards and automated red-teaming.
- Defenses for Prompt Injection: Unlike defenses that teach models to ignore instructions in user inputs, this work handles multiple instruction levels and their conflicts.The hierarchy distinguishes system, user, and third-party content rather than applying a single blanket rule.
- Over-refusal Evaluation: Over-refusal datasets test whether lower-privileged instructions aligned with higher-privileged ones remain followed, and performance is nearly baseline.The baseline usually follows all instructions, making it the comparison target for benign compliance.
- System-level Guardrails: Instruction-hierarchy defenses are complementary to system-level mitigations such as user approval or denial of API actions.The discussion anticipates more complex safeguards for agentic use cases.
- Automated Red-teaming: The training-data pipeline combines few-shot prompting, end-to-end attacker-LLM training, and context distillation for automated red-teaming.These methods generate adversarial training data for instruction-hierarchy models.
6 Conclusion & Future Work
The paper concludes that instruction hierarchy training improves resistance to adversarial manipulation, while identifying broader data scaling, modality coverage, architecture changes, and stronger adversarial training as future work.
- Conclusion: The proposed framework teaches language models to follow instructions while ignoring adversarial manipulation.The authors describe the current version as a dramatic improvement over the contemporary baseline.
- Future Work: The current hierarchy treats every instruction encountered during browsing or tool use as misaligned, motivating refinements to conflict handling.The authors also plan to extend the approach beyond text inputs to images and audio, and to explore specialized message-level embeddings.
- Limitations: Current models are likely still vulnerable to powerful adversarial attacks, so future work will add explicit adversarial training and assess robustness for high-stakes agentic applications.The paper does not claim that the present models are sufficiently robust for those applications.
A Additional Results
Additional results compare instruction-hierarchy training data with a prompting-based system-message baseline. The training approach performs much better, while combining training data with the system message can sometimes improve results further.
- The baseline teaches the instruction hierarchy by adding a System Message that explicitly defines it.
- Training data leads to much better robustness than the System Message approach.
- Combining instruction-hierarchy training data with the System Message sometimes produces the best results.
B Evaluation Details
The evaluation covers direct and indirect prompt injections, system-prompt extraction, jailbreaks, and over-refusal. It uses held-out attacks, adversarial datasets, tool and browsing scenarios, and GPT-4-based judgments across several tasks.
- Direct Prompt Injections: Direct prompt-injection evaluations place closed-domain tasks in the System Message and attacks in the User Message, then use GPT-4 to judge whether injections are ignored.
- Direct Prompt Injections: Open-domain direct-injection tests include Prompt Injection (Hijacking) and User Conflicting Instructions using held-out or adversarial data.
- Indirect Prompt Injections: Indirect-injection evaluations insert instructions into web results or external tool return values.
- System Prompt Extraction: System-prompt extraction tests measure recovered system-message unigrams, secret-password extraction, and Gandalf Game password extraction, including attacks unlike training examples.
- Jailbreaks: Jailbreak evaluations pair jailbreaks with unsafe requests and use GPT-4 to judge whether the model refused.
- Over-refusal Datasets: Over-refusal evaluations test whether models avoid refusing benign requests, including allowed prompts and borderline prompts.