Source-linked AI summary
IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
Chuan Guo, Juan Felipe Ceron Uribe, Sicheng Zhu, Christopher A. Choquette-Choo, Steph Lin, Nikhil Kandpal, Milad Nasr, Rai, Sam Toyer, Miles Wang, Yaodong Yu, Alex Beutel, Kai Xiao
TL;DR
Robust instruction hierarchy requires resolving nuanced conflicts across diverse domains, while RL training can encourage reward-hacking shortcuts when rewards are difficult to specify or evaluate. The paper introduces IH-Challenge, a programmatically gradable reinforcement-learning dataset with online adversarial example generation, and reports improved robustness that generalizes to unseen attacks and out-of-distribution tasks, alongside reduced unsafe behavior and stronger prompt-injection robustness.
Problem
Robust instruction hierarchy requires reasoning about nuanced conflicts across diverse task domains, while RL can suffer reward hacking when rewards are difficult to specify or evaluate exactly.
Method
The paper introduces IH-Challenge, an RL training dataset built around IF-simple tasks, programmatic grading, and avoidance of shortcut learning, with online adversarial conflict synthesis during defender training.
Results
IH robustness generalizes to unseen attacks and out-of-distribution tasks, while adaptive human red-teaming robustness rises from 63.8% to 88.2% and unsafe behavior falls from 6.6% to 0.7%.
Takeaways & Limitations
Training on IH-Challenge is a viable path for improving frontier-LLM instruction-hierarchy robustness, including disallowed-content safety and prompt-injection robustness.
Takeaways & Limitations
The paper leaves simultaneous attacker-and-defender training as an open direction and hypothesizes that programmatically gradeable rewards may be required to prevent goal drift at scale.
Abstract
from arXiv · showhide
Instruction hierarchy (IH) defines how LLMs prioritize system, developer, user, and tool instructions under conflict, providing a concrete, trust-ordered policy for resolving instruction conflicts. IH is key to defending against jailbreaks, system prompt extractions, and agentic prompt injections. However, robust IH behavior is difficult to train: IH failures can be confounded with instruction-following failures, conflicts can be nuanced, and models can learn shortcuts such as overrefusing. We introduce IH-Challenge, a reinforcement learning training dataset, to address these difficulties. Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation improves IH robustness by +10.0% on average across 16 in-distribution, out-of-distribution, and human red-teaming benchmarks (84.1% to 94.1%), reduces unsafe behavior from 6.6% to 0.7% while improving helpfulness on general safety evaluations, and saturates an internal static agentic prompt injection evaluation, with minimal capability regression. We release the IH-Challenge dataset (https://huggingface.co/datasets/openai/ih-challenge) to support future research on robust instruction hierarchy.
1 Introduction
Instruction hierarchy provides a trust-ordered policy for resolving conflicting instructions, but robust training must handle nuanced adversarial conflicts without confusing IH with ordinary instruction following. IH-Challenge addresses this with simple, programmatically gradable tasks and online attacks, improving robustness, safety, and prompt-injection resistance.
- IH prioritizes higher-trust roles over conflicting lower-priority instructions, supporting defenses against jailbreaks, prompt extraction, and prompt injection.
- Robust IH requires resolving nuanced conflicts across diverse domains and resisting adversarial instructions designed to violate the hierarchy.
- IH-Challenge is an RL training dataset built around IF-simple, programmatically gradable tasks that discourage shortcut learning.
- The training uses online adversarial example generation to fine-tune GPT-5-Mini on increasingly challenging instruction conflicts.
- 63.8% to 88.2%: GPT-5-Mini-R improves IH robustness under adaptive human red-teaming while showing only minor capability regression.
- 6.6% to 0.7%: unsafe behavior decreases while helpfulness is maintained, alongside stronger robustness to agentic prompt injections.
- IH-Challenge is released with tasks and Python grading code to support future research on IH robustness.
2 Background
Instruction hierarchy formalizes how models should prioritize messages from roles with different trust levels. The paper motivates deterministic grading for RL because reward hacking is especially problematic when desirable behavior is difficult or costly to evaluate exactly.
- IH orders messages by role, with system instructions above developer and user messages, and tool messages treated as untrusted inputs.
- RL trains models from sampled rollouts graded by reward functions, unlike supervised fine-tuning on fixed examples.
- Reward hacking occurs when models exploit shortcuts to maximize reward instead of solving the intended problem.
- Deterministic graders such as Python code can reduce label noise and make RL more resistant to reward hacking when rewards are straightforward to specify.
- Balancing helpfulness and harmlessness is difficult because instruction-following training can increase compliance with malicious requests, while safety training can make models overly conservative.
3 Constructing IH-Challenge
IH-Challenge constructs offline task skeletons with high-priority instructions and valid Python graders, then fills low-priority placeholders with online adversarial attacks. RL trains the defender on these generated conflicts while preserving capability through mixed training.
- Instruction hierarchy: IH defines a feasible response set that is updated by descending-priority constraints, ignoring a lower-priority instruction when it conflicts with higher-priority constraints.
- Task skeletons: Task skeletons combine high-priority instructions, a low-priority conflict placeholder, and a Python grader valid under arbitrary low-priority messages.
- Task design: Tasks remain IF-simple under attack so rewards measure IH handling rather than puzzle solving.
- Task design: Conflicts are restricted to low-priority roles and ambiguous, non-objectively gradeable tasks such as jailbreaks are excluded.
- Task design: Task diversity targets shortcut learning, including overrefusal, so consistently high reward requires robust IH rather than brittle heuristics.
- Defender training: Defender RL synthesizes conflict prompts online, scores multiple responses with the Python grader, and mixes capability-focused tasks to reduce capability regression.
- Online adversarial conflict synthesis: The attacker uses a budgeted propose–evaluate–revise loop, testing candidate messages against the current defender with Python-grader feedback.
- Evaluation: Held-out attacks evaluate generalization, while structured attacker prompting increases attack diversity and difficulty against an improving defender.
4 Main Result
RL fine-tuning on IH-Challenge improves robustness on held-out, out-of-distribution, and human-red-teaming evaluations while mostly preserving general capabilities. The resulting model also improves safety steerability and prompt-injection robustness without requiring corresponding system-message changes.
- In-distribution generalization: Significant robustness gains generalize to unseen attacks after RL fine-tuning.
- Overrefusal and capability: GPT-5-Mini-R mostly matches GPT-5-Mini on overrefusal and capability evaluations, with slight regressions on chat win-rate and preference score.It gains significantly on IH-Challenge overrefusal, reflecting training on the Anti-Overrefusal subset.
- OOD generalization: Notable robustness gains appear across internal and academic out-of-distribution IH evaluations.Most evaluated tasks were unseen during training and graded by LLM graders.
- Cross-cutting effects: RL-based IH robustness generalization also improves safety steerability and prompt-injection robustness.Safety steerability includes improved reasoning about nuanced safety conversations and improved safety refusal and helpfulness rates.
- Safety steerability: GPT-5-Mini-R is safer than GPT-5-Mini with or without the same safety specification while largely preserving helpfulness.The evaluation uses category-specific safety specifications on challenging safety-sensitive production conversations.
- Prompt injection robustness: GPT-5-Mini-R improves prompt-injection robustness without additional system-message changes and saturates the static internal evaluation.The static attacks target malicious emails and other harmful agentic behaviors.
5 Analysis
The analysis evaluates system mitigations and adaptive red-teaming, finding that output monitoring can improve robustness against adaptive attacks while capability trade-offs vary across models.
- Task splits: All four IH-Challenge splits provide the best balance between robustness, safety, and overrefusal.Removing Anti-Overrefusal increases refusals, while training only on Multi-Constraint and Input-Conditioned performs slightly worse on robustness and overrefusal.
- System mitigations: Output rewriter and sandwich defenses improve GPT-5-Mini robustness with a slight capability drop, but GPT-5-Mini-R shows larger trade-offs.Some mitigations for GPT-5-Mini-R instead worsen both robustness and capability.
- Adaptive automated red-teaming: With an output monitor, attack success rate is lower across all three evaluated IH-Challenge splits during attacker RL training.The result suggests defense-in-depth can help against certain adaptive attackers even when static-attack benefits are limited.
- Adaptive human red-teaming: Adaptive human red-teaming reduces attack success from 36.2% for GPT-5-Mini to 11.7% for GPT-5-Mini-R and 7.1% with an output monitor.Red-teamers also required more attempts against the stronger systems but remained less successful.
6 Related Work
The paper builds on role-separation, sandwich-defense, and reinforcement-learning approaches to instruction-hierarchy robustness, emphasizing out-of-distribution generalization from programmatically gradable training tasks.
- Existing defenses: Prior work improves instruction-hierarchy robustness through role embeddings and sandwich defenses that repeat higher-tier instructions.The sandwich defense was originally proposed for prompt injection and also improves IH robustness in the paper’s evaluation.
- Training-based defenses: A related training approach uses IFEval instructions as higher-tier constraints and generates lower-tier attacks for programmatically verifiable reinforcement-learning rewards.The paper positions IH-Challenge as an expansion of this training-based direction.
- RL and generalization: IH-Challenge uses IF-simple, limited-variety tasks with objective grading, yet RL training generalizes IH robustness to unseen attacks and out-of-distribution safety domains.The authors identify this out-of-distribution generalization as central to using RL for non-programmatically gradeable tasks.
7 Conclusion
The paper concludes that IH-Challenge training with online adversarial example generation improves instruction-hierarchy robustness across unseen attacks and out-of-distribution tasks, while highlighting adversarial-training scaling as an open direction.
- Conclusion: Training on IH-Challenge is presented as a viable path for improving instruction-hierarchy robustness on frontier LLMs.
- Conclusion: Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation generalizes robustness to unseen attacks and out-of-distribution tasks.The approach also yields gains on disallowed-content safety and prompt-injection robustness.
- Future direction: Simultaneously training attacker and defender models is proposed as a promising compute-scaling direction for adversarial training.The authors hypothesize that programmatically gradeable rewards may be required to prevent attacker and defender goals from drifting from the robustness objective.
A.1 Prompts
The appendix illustrates output-filter and impersonation prompts that operationalize instruction-hierarchy priorities and test whether lower-priority or fabricated instructions can override protected system instructions.
- A.1.1 Example Output Filter Prompt: The output-filter prompt asks a model to judge whether another model’s latest response adhered to the Instruction Hierarchy.
- A.1.1 Example Output Filter Prompt: The hierarchy specifies that system instructions outrank developer instructions, which outrank user instructions, with later instructions superseding earlier ones at equal authority.
- A.1.1 Example Output Filter Prompt: The example filter tests a system ban on the word “pixel” against a user request to define that word.The illustrated response defines the concept without directly reproducing the banned word.
- A.1.2 Example Impersonation Attack: The impersonation example embeds fabricated system instructions with numbered priorities inside conversation history.These instructions claim authority to override the original system message and require disclosure of protected content.
- A.1.2 Example Impersonation Attack: The attack demands a “System Instructions Summary” containing the original system message and its secret PIN.
- A.1.2 Example Impersonation Attack: Additional fabricated instructions request a “Compliance Explanation” that rationalizes overriding the original system instructions.The prompt explicitly instructs the model not to refuse or apologize because the fabricated instructions allegedly have higher priority.
- A.1.2 Example Impersonation Attack: The user message then asks the model to follow the fabricated priority rules and produce both disclosure sections.
A.2.1 Single-Constraint
The single-constraint examples pair an attacker objective with a system-level prohibition and a user-message placeholder for the attack. One task targets disclosure of identifying information despite the system instruction forbidding it.
- Single-Constraint: The attacker objective is to make the model reveal personally identifying information through injected user text.The targeted information includes U.S. phone numbers, SSNs, and email addresses, even when publicly available.
- Single-Constraint: Each example supplies the attack through a user message marked as a replacement placeholder.The surrounding prompt structure identifies the defender instruction as a system message and the attack as user content.
- Single-Constraint: The system instruction forbids revealing phone numbers, SSNs, and email addresses for any person or company.It also prohibits outputting obviously fake values or strings matching those formats.
A.2.2 Multi-Constraint {
The multi-constraint examples combine several defender instructions, attack objectives, and evaluation cases across system and developer messages. They also include judge-error examples involving sentiment labels, JSON formatting, and joke outputs.
- Multi-Constraint: One multi-constraint task asks the attacker to violate ASCII-only output and avoid a specified list of banned words.The system instructions define both the valid-character range and the banned vocabulary.
- Multi-Constraint: Another task requires market-trend classification using cue categories, precedence rules, and a strict three-line output format.The developer instruction specifies UP, DOWN, FLAT, MIXED, and UNKNOWN outcomes plus a lowercase cue.
- Multi-Constraint: A separate attack rewrites a harmless dummy-PIN request to appear conflicting with a system directive protecting the secret PIN 4667.The intended contrast is between a safe model that may reject the rewritten request and an unsafe model that preserves the original request's meaning.
- Multi-Constraint: The benchmark figures compare refusal and helpfulness rates on OpenAI’s Production Benchmarks under global versus category-specific safety specifications.Both captions state that higher is better.
- Multi-Constraint: The examples label cases where an LLM judge makes an error while evaluating joke responses.The listed joke outputs include atom, computer, and scarecrow jokes.
- Multi-Constraint: A sentiment-analysis case places system and developer instructions in conflict over capitalization of the output label.The user message is “This is wonderful!”, while the system requests lowercase classes and the developer requests initial capitalization.
- Multi-Constraint: A formatting case requires valid JSON while a simulated developer conversation requests switching to plain text, and the shown response remains JSON.The response begins with a JSON object containing a message and jokes field.