PaperlayerView exact revision on arXiv

Source-linked AI summary

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, Alex Beutel

arXiv:2404.13208v1cs.CRcs.CLcs.LG

TL;DR

LLMs can treat trusted and untrusted instructions as equally authoritative, allowing adversaries to override higher-priority instructions. The paper fine-tunes GPT-3.5 Turbo on generated examples that teach an instruction hierarchy. Robustness improves across evaluations, including unseen attacks, while generic capabilities remain comparable to the baseline.

  • Problem

    Existing LLMs lack distinctions between instruction privileges across system, user, and third-party text, allowing adversarial prompts to override higher-priority instructions.

    Source: §1 · pp. 1–2

  • Method

    The authors fine-tune GPT-3.5 Turbo on generated examples that teach models to follow aligned instructions across hierarchy levels and ignore conflicting lower-priority instructions.

    Source: §3.2 · p. 4, §4 · p. 7

  • Results

    Robustness improves across all evaluations, including unseen attacks, with system prompt extraction defense improved by 63% and generic capabilities comparable to the baseline.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    The generalization results suggest that models can internalize instruction priorities, becoming safer and more controllable even for unseen prompts.

    Source: §4 · p. 7

    The models likely remain vulnerable to powerful adversarial attacks, and robustness sufficient for high-stakes agentic applications remains an open question.

    Source: §6 · p. 9

Abstract

from arXiv · show

Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities.

1 Introduction

The paper proposes training LLMs to prioritize system messages over user messages, and user messages over third-party content. Its approach improves robustness across evaluated attacks, including held-out attacks, while introducing some benign-query over-refusals but otherwise preserving generic capabilities.

1 Introduction

  • Adversarial instructions threaten agentic applications by enabling unsafe actions, bypassing developer restrictions, exposing company IP, and compromising users’ private data or sessions.The introduction illustrates the risk with a hypothetical prompt injection that causes an email assistant to exfiltrate private emails.Source: §1 · p. 1
  • The proposed hierarchy addresses missing instruction privileges by prioritizing system messages over user messages, and user messages over third-party content.The authors argue that insufficient separation between message types lets adversaries override higher-level instructions.Source: §1 · pp. 1–2
  • Models should follow aligned lower-privileged instructions, ignore conflicting lower-privileged instructions when possible, and refuse when proceeding is impossible.Alignment depends on compatibility with higher-privileged instructions, rather than merely where an instruction appears.Source: §1 · p. 2
  • Synthetic data generation and context distillation (Askell et al., 2021; Snell et al., 2022) teach hierarchy-following through aligned compositions and red-teamer-generated misaligned attacks.Aligned requests are decomposed across hierarchy levels while preserving the original target response; misaligned examples teach models to act ignorant of lower-level instructions. These examples and generic instruction-following data fine-tune GPT-3.5 Turbo using supervised fine-tuning and RLHF.Source: §1 · p. 2
  • 63% improvement in defense against system prompt extraction and over 30% improvement in jailbreak robustness accompany robustness gains across all evaluations.Evaluation uses open-sourced and novel benchmarks, with generalization to held-out attacks not directly modeled during data generation. Models sometimes ignore or refuse benign queries, but generic capabilities otherwise remain intact.Source: §1 · p. 2

2 Background: Attacks on LLMs

LLM applications combine developer instructions, user inputs, and third-party context, creating opportunities for conflicting instructions. Prompt injections subvert application intent, jailbreaks bypass trained safety behavior, and system-message extraction attacks reveal privileged prompts or secrets.

The Anatomy of an LLM

  • Modern chat LLMs distinguish system messages, user messages, model outputs, and tool outputs using special tokens.Only application developers provide system messages, which specify instructions, safety guidelines, constraints, and available tools. Applications may involve a developer controlling instructions and execution flow, a main user, and third-party inputs consumed as context.Source: §2 · pp. 2–3

Prompt Injections

Jailbreaks

  • Jailbreaks aim to bypass an LLM’s trained safety behavior and often do not conflict with its previous instructions.Attack variants can enable malicious tasks such as spam, misinformation, or pornographic content (Schulhoff et al., 2023; Zou et al., 2023).Source: §2 · p. 3

System Message Extraction

  • System-message extraction attacks seek to reveal the entire system message or a specific secret, even when developers explicitly prohibit disclosure.System messages may contain business logic, passwords, or safety instructions, and knowing the prompt can aid other attacks (Toyer et al., 2024). Extraction attacks are straightforward to execute even under instructions against repeating the prompt (Perez & Ribeiro, 2022; Zhang & Ippolito, 2023; Schulhoff et al., 2023).Source: §2 · p. 3

3 The Instruction Hierarchy

The proposed instruction hierarchy makes models defer to higher-privileged instructions when instructions conflict. Training combines aligned instructions across hierarchy levels while teaching models to ignore misaligned instructions or refuse when necessary.

3 The Instruction Hierarchy

  • The instruction hierarchy resolves conflicts by prioritizing higher-privileged instructions over lower-privileged instructions.The authors identify the absence of such a hierarchy as an underlying cause of instruction-based attacks.Source: §3 · p. 4

3.1 Overview of Ideal Model Behavior

  • Models should follow aligned lower-level instructions, ignore conflicting or orthogonal instructions when possible, and otherwise refuse.Alignment depends on whether lower-level instructions share higher-level constraints, rules, or goals.Source: §3.1 · p. 4

3.2 Training Data Generation for Different Attacks

  • Context synthesis distributes aligned instructions across hierarchy levels; context ignorance trains models to answer as though misaligned lower-level instructions were absent.Data generation balances injection resistance against overrefusal, preserving the ability to follow aligned lower-priority instructions.Source: §3.2 · p. 4
  • Open-domain training decomposes compositional requests for aligned examples and generates adversarial user queries against system constraints for misaligned examples.Misaligned targets either ignore the injection or satisfy the constraint, with constraint-satisfying responses collected through repeated sampling and separate LLM checks. Models learn refusal when proceeding is impossible.Source: §3.2 · pp. 5–6
  • Closed-domain training assumes no aligned input instructions and uses context distillation to teach models to treat embedded instructions as data.Ground-truth responses are generated with an explicit instruction to treat embedded commands as data. A GPT-4 grader filters successful injections before training without that extra system instruction.Source: §3.2 · p. 6
  • Indirect-injection training treats browsing and tool-use instructions as misaligned, pairing adversarial search-result injections with the original ground-truth answers.An RL-trained red-teamer is rewarded for confusing the base model. Training excludes injections from non-browsing tools and all jailbreak data to test generalization.Source: §3.2 · p. 6
  • System-message training refuses extraction requests while answering basic capability questions; private-information and password extraction attacks are withheld to test generalization.Misaligned training targets use context ignorance by treating the system message as unseen; aligned examples use synthetic questions about the system message.Source: §3.2 · p. 6

4 Main Results

The instruction hierarchy improved safety across all main evaluations and generalized to attack types excluded from training, while maintaining comparable generic capabilities. Models followed non-conflicting instructions almost as well as the baseline on most evaluations, but regressed on two tasks.

Experiment Setup

  • GPT-3.5 Turbo retained comparable generic capabilities to the baseline on evaluations including TriviaQA, LAMBADA, and HellaSwag.Both models used supervised fine-tuning and reinforcement learning from human feedback (Ouyang et al., 2022), selecting checkpoints by validation accuracy. The evaluation suite covered in-domain attacks, generalization attacks, and over-refusal, with one-standard-deviation error bars and higher-is-better metrics.Source: §4 · p. 7

Main Results

  • Up to 63% robustness improvement was reported with the instruction hierarchy, which improved safety across all main evaluations.The comparison baseline was similarly fine-tuned but trained without instruction-hierarchy examples.Source: §4 · p. 7

Generalization Results

  • Up to 34% robustness improvement extended to evaluation criteria explicitly excluded from training.These included jailbreaks targeting unsafe outputs, system-message password extraction, and prompt injections through tool use.Source: §4 · p. 7

Main Results

  • Qualitative adversarial tests demonstrate treating user input as data or refusing attacks, including withholding a password that the baseline revealed.Table 2 presents three correct responses: one treats user input as data rather than instructions, and two refuse.Source: Table 2 · p. 8, §4 · p. 8

Over-refusal Results

  • Non-conflicting instruction following nearly matched the baseline on most over-refusal evaluations, with regressions on System Message Probing Questions and Jailbreakchat with Allowed Prompts.These evaluations tested benign instructions and safe boundary cases resembling attacks; both regression tasks were adversarially constructed to expose instruction-hierarchy sensitivities.Source: §4 · p. 8

5 Discussion & Related Work

The approach extends prompt-injection defenses to multiple instruction levels while retaining compatible lower-level instructions. It complements system-level guardrails and uses automatically generated adversarial training data.

Defenses for Prompt Injection

Overrefusal

  • On adversarial over-refusal evaluations, the models follow non-conflicting lower-privileged instructions nearly as well as the baseline model.These evaluations test whether lower-privileged instructions are followed when aligned with higher-privileged ones; the baseline usually follows all instructions.Source: Figure 4 · p. 9

System-level Guardrails

  • Model-based attack mitigation complements system-level guardrails, such as requiring users to approve API calls, with more complex safeguards envisioned for agentic applications.The authors draw an analogy to existing Internet protections, including unsafe-website detection and phishing classifiers.Source: §5 · p. 9

Automated Red-teaming

  • The work generates adversarial training data through few-shot prompting, end-to-end training of attacker LLMs, and context distillation.Related work generates red-teaming data with LLMs (Perez et al., 2022; Ganguli et al., 2022) or stronger adversaries through gradient-based transfer attacks (Wallace et al., 2019; Zou et al., 2023; Geiping et al., 2024).Source: §5 · p. 9

6 Conclusion & Future Work

The instruction hierarchy teaches language models to follow instructions while ignoring adversarial manipulation, with the authors reporting a dramatic improvement over current LLM behavior. Future work targets better conflict handling, multimodal inputs, architectural changes, and stronger adversarial robustness.

Conclusion & Future Work

  • The instruction hierarchy substantially improves resistance to adversarial manipulation while preserving instruction following.The authors expect larger-scale data collection, guided by their behavior taxonomy and over-refusal evaluations, to improve performance and refine refusal boundaries.Source: §6 · p. 9
  • Conflict handling needs refinement because current models are trained never to follow instructions during browsing or tool use.Future work will revisit this blanket treatment of instructions.Source: §6 · p. 9
  • Planned multimodal work includes testing generalization to images and audio and creating multimodal instruction-hierarchy data.These modalities can also contain injected instructions.Source: §6 · p. 9
  • Architectural changes, such as specialized embeddings for messages at different priority levels, will be explored to better instill the instruction hierarchy.Source: §6 · p. 9
  • Current models likely remain vulnerable to powerful adversarial attacks, motivating explicit adversarial training and investigation of robustness sufficient for high-stakes agentic applications.Whether LLMs can achieve sufficient robustness for these applications remains an open question.Source: §6 · p. 9

A Additional Results

Instruction-hierarchy training data produces much better robustness than explicitly defining the hierarchy in a system message; combining training with that message sometimes improves results further.

System Message Baseline

  • Adding the hierarchy-defining system message to the final trained model sometimes provides further robustness gains.Combining instruction-hierarchy data with the system message achieves the best results in some cases.Source: A Additional Results · p. 12, Figure 5 · p. 12
  • The prompting baseline uses a system message that explicitly defines the instruction hierarchy, with the baseline message identified in Table 3.Source: A Additional Results · p. 12, Table 3 · p. 12
  • Instruction-hierarchy training data provides much better robustness than a system-message prompting baseline.The baseline explicitly defines the instruction hierarchy rather than teaching it through training data.Source: Figure 5 · p. 12, A Additional Results · p. 12

B Evaluation Details

The evaluation covers direct and indirect prompt injections, system-prompt extraction, jailbreaks, and over-refusal, using attack-specific measures and GPT-4 judgments.

Direct Prompt Injections

  • Direct-injection evaluations place closed-domain tasks or defender prompts in system messages and attacks in user messages, testing instruction resistance and hijacking, while separate user-instruction evaluations measure refusal and non-refusal.Closed-domain evaluation holds out automated red-team injections and uses GPT-4 to judge whether injected instructions are not followed (higher is better). TensorTrust Prompt Hijacking (Toyer et al., 2024) measures how often “Access Granted” appears in outputs; evaluations using held-out synthetic system prompts use GPT-4 to judge refusal for unreasonable user instructions and non-refusal for reasonable instructions (higher is better for non-refusal).Source: B Evaluation Details · p. 13

Indirect Prompt Injections

  • Indirect-injection evaluations insert manually crafted instructions into browsing results and held-out red-teamer-generated instructions into external-tool return values.The two datasets separately test injections delivered through browsing and tool outputs.Source: B Evaluation Details · p. 13

System Prompt Extraction

  • System-prompt extraction evaluations measure prompt and password leakage, including two attack evaluations unlike training examples, alongside an over-refusal check.TensorTrust Robustness (Toyer et al., 2024) measures disclosure of more than half the system-prompt unigrams; password tests use original TensorTrust defender prompts and Gandalf Game (Lakera AI, 2023) attacks targeting “PLANETARY” under an explicit secrecy instruction. Both password evaluations differ from training examples, while GPT-4 judges refusal on held-out synthetic questions about the system prompt.Source: B Evaluation Details · p. 13

Jailbreaks

  • Jailbreak evaluations place jailbreaks in system messages, pairing Jailbreakchat attacks and known successful ChatGPT jailbreaks with unsafe requests for GPT-4 assessment.For Jailbreakchat with unsafe prompts, GPT-4 specifically judges whether the model refused.Source: B Evaluation Details · p. 13

Over-refusal Datasets

  • Additional over-refusal evaluations measure GPT-4-judged non-refusal on benign requests paired with Jailbreakchat attacks and benign prompts that may appear malicious.Higher non-refusal rates are better on both datasets.Source: B Evaluation Details · p. 13

Related papers

Loading 2404.13208v1…