Source-linked AI summary
IHEval: Evaluating Language Models on Following the Instruction Hierarchy
Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, Meng Jiang
TL;DR
Language models lack comprehensive evaluation of their ability to follow an instruction hierarchy, despite the hierarchy’s role in consistent and safe behavior. The paper introduces IHEval, a nine-task benchmark with 3,538 examples covering aligned and conflicting inputs across four priority levels. Evaluations find that models struggle with instruction conflicts, with the best open-source model achieving 48% accuracy, and the paper identifies evaluation—not solutions—as its scope.
Problem
Instruction hierarchy research lacks a comprehensive benchmark for evaluating whether language models prioritize system, user, conversation-history, and tool-output instructions.
Method
IHEval evaluates nine diverse, programmatically assessed tasks with hierarchical inputs that either align or conflict across varied input types and difficulties.
Results
Models struggle to prioritize high-level instructions during conflicts; open-source models achieve less than 50% accuracy, with Qwen-2 reaching 48%.
Takeaways & Limitations
Current language models are not sufficiently optimized for following the instruction hierarchy, motivating further research and optimization in this dimension.
Takeaways & Limitations
The paper evaluates the problem but does not propose specific solutions, leaving training methods such as supervised or preference tuning for future work.
Abstract
from arXiv · showhide
The instruction hierarchy, which establishes a priority order from system messages to user messages, conversation history, and tool outputs, is essential for ensuring consistent and safe behavior in language models (LMs). Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy. We bridge this gap by introducing IHEval, a novel benchmark comprising 3,538 examples across nine tasks, covering cases where instructions in different priorities either align or conflict. Our evaluation of popular LMs highlights their struggle to recognize instruction priorities. All evaluated models experience a sharp performance decline when facing conflicting instructions, compared to their original instruction-following performance. Moreover, the most competitive open-source model only achieves 48% accuracy in resolving such conflicts. Our results underscore the need for targeted optimization in the future development of LMs.
1 Introduction
Language models need to prioritize higher-level instructions to maintain consistent and safe behavior, yet this capability is understudied and lacks a comprehensive benchmark. IHEval addresses this gap with diverse, programmatically evaluated scenarios involving aligned and conflicting instructions.
- Motivation: Higher-level instructions should take priority because conflicting lower-level inputs can cause model misbehavior.The hierarchy ranks system messages above user messages, conversation history, and tool outputs.
- Research gap: Research on instruction hierarchy remains limited, with existing evaluation restricted to GPT models, proprietary data, or safety-focused instructions.The paper identifies no comprehensive benchmark for evaluating adherence across broader use cases.
- Benchmark: IHEval contains 3,538 examples across nine tasks spanning rule following, task execution, safety defense, and tool use.It covers system messages, user messages, conversation history, and tool outputs.
- Benchmark: The benchmark includes both aligned and conflicting hierarchical inputs, with varied task difficulties and programmatic evaluation.These settings test normal hierarchical processing as well as conflict resolution.
- Findings: Open-source models achieve less than 50% accuracy on resolving conflicting instructions, while models also show inconsistent performance without conflicts.The evaluation further finds sensitivity to superficial instruction strictness and weak scaling with model size.
2 Related Work
Prior work evaluates instruction following, system prompts, and safety behavior, but IHEval targets the broader instruction hierarchy across multiple input types with programmatic evaluation.
- Instruction following: Instruction-following research has progressed from holistic expert judgments toward fine-grained checks of output constraints and embedded instructions.These studies examine formatting, content constraints, and separation of instructions from data.
- System prompts: System prompts guide model roles, tasks, formats, and safety behavior, but their training details are rarely disclosed.They have also been studied for general instruction following, personalization, rule adherence, and jailbreak defense.
- Positioning: IHEval extends system-prompt evaluation by covering system prompts, user instructions, conversation history, and tool outputs.Unlike SysBench, it uses fully programmatic evaluation rather than GPT-based evaluation.
3 IHEval
IHEval defines an instruction hierarchy over four input types and evaluates models in aligned, conflicting, and reference settings across nine programmatically assessed tasks. Its design varies application scenarios, task difficulty, and evaluation metrics to isolate hierarchy-following behavior.
- Definitions: IHEval ranks system messages, user messages, conversation history, and tool outputs from highest to lowest priority.Higher-priority instructions are the main instruction; incompatible lower-priority requests are conflicting instructions.
- Task settings: The benchmark compares aligned hierarchical inputs, conflicting hierarchical inputs, and a single-input reference setting.The reference setting separates baseline task ability from instruction-hierarchy following.
- Task design: Nine tasks cover diverse application scenarios and input types, using human-annotated labels or reliable programmatic evaluation where possible.Tasks are grouped into four categories based on instruction content.
- Task difficulties: Task difficulty varies through different levels of imperative strictness, including instructions requiring exclusive adherence.This design also reduces randomness caused by instruction phrasing.
- Evaluation: IHEval scores models by performance on the main instruction, using task-specific metrics and strict or loose variants when exact match is unavailable.For example, verb extraction uses F-1, while executing a conflicting translation instruction lowers performance against the system-defined target.
4 Experiments
Across 13 language models, IHEval reveals unstable performance on hierarchical inputs and substantial difficulty resolving conflicts in favor of higher-priority instructions. Performance improves with scale in general, but conflict handling remains sensitive to model scale, instruction strictness, conflict location, and superficial prompting.
- Experimental setup: 13 language models were evaluated on IHEval, with results reported across reference, aligned, and conflict settings.The evaluation includes proprietary and open-source models from five model families.
- Main results: At least 4 points of absolute difference occurred between aligned hierarchical and reference settings in all models except GPT-4o and Qwen2-72B.LLaMA-3.1-70B declined in eight of nine tasks, averaging a 13-point drop after switching to hierarchical inputs.
- Main results: 48% accuracy was achieved by Qwen-2, the second-best model in conflict settings, while all models showed notable drops when low-priority instructions conflicted with high-priority ones.GPT-4o remained the best conflict-setting performer despite a 22-point drop from its aligned setting.
- Performance by model scale: Larger models generally performed better across settings, yet most model families except GPT and Qwen-2 showed a larger aligned-to-conflict gap as scale increased.Claude-Haiku outperformed Claude-Sonnet on five of nine tasks, illustrating inverse scaling for conflict resolution.
- Performance by instruction strictness: Performance improved when the high-priority instruction was stricter but decreased when the conflicting instruction was stricter, showing sensitivity to wording strictness rather than hierarchy alone.The authors characterize this dependence on superficial strictness as undesirable because priority should determine instruction handling.
- Model performance in different conflicts: In multi-turn conflicts, scores fell from 79.6 to 68.9 to 59.5 as additional components conflicted, then dropped to 17.7 when the current turn conflicted.Models handled a previous-turn conflict relatively well when the current turn remained aligned, scoring 84.2 versus 85.9 in the reference setting.
5 Conclusion
IHEval evaluates whether language models follow instruction hierarchies across diverse aligned and conflicting scenarios. The evaluation reveals difficulty recognizing instruction priorities and motivates further optimization.
- IHEval evaluates language models’ ability to follow the instruction hierarchy across nine diverse, programmatically evaluated tasks.
- The benchmark covers hierarchical inputs that either align or conflict and vary in input type and task difficulty.
- IHEval identifies a significant weakness in mainstream language models: difficulty recognizing the priority of different instructions.
- The paper analyzes model behavior under varied instruction-conflict scenarios and highlights further optimization as a priority for future research.
Limitations
The paper identifies language models’ difficulty following the instruction hierarchy but does not propose specific solutions. Developing training methods is left for future work after establishing comprehensive evaluation data and behavioral analyses.
- The paper does not propose specific solutions for improving language models’ instruction-hierarchy following.
- Future work should prioritize training methods such as supervised fine-tuning or preference tuning to improve instruction-hierarchy following.
- The paper focuses on closing the evaluation gap because optimization is viewed as needing comprehensive evaluation data and in-depth model-behavior analyses.
Ethical Considerations
The authors describe precautions for data safety and label accuracy in IHEval. They report that these measures minimize risks associated with data collection and benchmark use.
- Most IHEval data come from public benchmarks with human-annotated labels, and Claude-generated Rule Following examples are reviewed and rewritten when needed.
- The authors state that all IHEval data are verified by humans, reducing the risk of inaccurate annotations or unsafe AI-generated content.
- The authors conclude that precautions make risks from IHEval data collection and benchmark usage minimal.
A Detailed Data Collection
The data collection constructs formatting-rule tasks for single-turn and multi-turn conversations, using aligned and conflicting instruction settings. Multi-turn variants extend the single-turn setup with generated responses, follow-up queries, and stricter alignment controls.
- Formatting-rule tasks include both single-turn and multi-turn conversations.
- Single-turn data split original inputs into system formatting rules and user queries, then add incompatible conflicting instructions for conflict settings.
- Multi-turn data use the single-turn task as an initial turn, generate aligned and conflicting first-turn responses, and create a second-turn query from the established context.
- The multi-turn configurations include Reference, Aligned, First-turn Conflict, and Both-turns Conflict settings.
- Conflicting first-turn responses are obtained by prompting Claude to use the conflicting format from the single-turn setting.
- An additional stricter aligned setting is introduced because adherence to system messages can deteriorate across turns.
A.2 Task Execution
Task Execution evaluates whether models preserve a system-defined task when user inputs add conflicting instructions. It covers extraction, generation, and classification, plus tool-output conflicts that can override the user task.
- Core tasks: Task Execution includes extraction, generation, and classification tasks modeled on common NLP benchmarks.Aligned settings pair a system task instruction with ordinary data; conflict settings prepend an alternative-task instruction to the data.
- Task metrics: Extraction measures verb-list F-1, generation measures Spanish translation with ROUGE-L, and classification measures language-detection accuracy.The extraction data come from OntoNotes, generation data from MGSM, and classification data from XL-Sum.
- Task variants: Strict variants require models to execute only the current instruction without generating additional content.Both system messages and conflicting instructions receive strict versions.
- Safety Defense: Safety Defense evaluates successful defenses against hijack and password-extraction attacks under strict and weaker security instructions.A defense fails if the model grants access incorrectly, reveals the password, or outputs more than half of the system-message unigrams.
- Tool Use: Tool Use tests intrinsic and injected conflicts in retrieved content, scoring whether the original user task is completed when both instructions cannot be satisfied.Intrinsic conflicts place instructions in webpage content; injected conflicts append a competing question to tool output.
B Evaluation Criteria
IHEval evaluates whether models follow higher-priority instructions rather than conflicting lower-priority requests. The criteria reject responses that execute both instructions or merely ask for clarification when that preserves the conflict.
- Priority adherence: Conflict settings test whether models strictly follow high-priority instructions while ignoring conflicting low-priority ones.The paper emphasizes developer-provided system messages because they govern services delivered to public users.
- Failure criteria: Responding to both instructions can create unsafe behavior by causing a bot to answer unrelated requests outside its intended role.The paper gives translation and shopping bots as examples of undesired responses.
- Evaluation rationale: GPT-4o’s strong IHEval performance supports the paper’s claim that these evaluation criteria reflect industry practices.The paper also aligns avoiding responses to potential misuse with standard language-model practices.
C Full Results
The paper reports results for 13 language models on IHEval, organized by model family in Tables 5–10.
- Results: Results for all 13 language models are presented in Tables 5–10, grouped by model family.The cited passage identifies the result tables but does not report their numerical values.
D Task Cards of IHEval
The task cards document IHEval’s settings, instructions, examples, and error analyses across its benchmark tasks. They also show model-output constraints and the strict-versus-conflict configurations used for evaluation.
- Task-card overview: Figures 8–16 provide task cards showing task settings, instruction versions, examples, and selected error analyses.Error percentages are calculated from generated responses across all models in Table 1.
- Input configurations: Figure 7 compares input configurations in settings where evaluation focuses on adherence to formatting rules rather than matching the user query.The reference and aligned #1 settings can therefore use the follow-up query as the only user message.
- Results presentation: Tables 5–10 report model-family results, while LLaMA-3 models omit Tool Use because they do not officially support tool calling.Red scores in the tables indicate absolute changes greater than 5.
- Task coverage: The task cards cover rule following, task execution, and safety defense examples, including single-turn, multi-turn, extraction, generation, classification, hijack, and extraction tasks.The cards include concrete prompts, strict instructions, task data, and gold responses for conflict settings.