PaperlayerView exact revision on arXiv

Source-linked AI summary

Many-Tier Instruction Hierarchy in LLM Agents

Jingyu Zhang, Tianjian Li, William Jurayj, Hongyuan Zhan, Benjamin Van Durme, Daniel Khashabi

arXiv:2604.09443v3cs.CLcs.AI

TL;DR

Existing instruction-hierarchy evaluations use only a few fixed role-based tiers, leaving unresolved whether agents can reliably resolve conflicts across many privilege levels. ManyIH introduces dynamically assigned privilege values and benchmarks this capability across 853 tasks with up to 12 tiers, finding frontier models achieve at most ~40% accuracy and degrade as tiers increase.

  • Problem

    Existing instruction-hierarchy evaluations largely use fixed role-based privilege levels, leaving adequacy for agents facing arbitrarily many tools, skills, and sub-agents unresolved.

  • Method

    ManyIH dynamically assigns privilege values to instructions and evaluates conflict resolution with MANYIH-BENCH, an 853-task benchmark spanning coding and instruction-following with up to 12 tiers.

  • Results

    At most ~40% accuracy is achieved by frontier models, with performance degrading monotonically as privilege tiers increase.

  • Takeaways & Limitations

    ManyIH establishes fine-grained, scalable instruction-conflict resolution as a distinct, unsolved capability for current language models.

  • Takeaways & Limitations

    The benchmark assumes privilege is assigned by a trusted party and does not study adversarial privilege manipulation.

Abstract

from arXiv · show

Large language model agents receive instructions from many sources-system messages, user prompts, tool outputs, other agents, and more-each carrying different levels of trust and authority. When these instructions conflict, agents must reliably follow the highest-privilege instruction to remain safe and effective. The dominant paradigm, instruction hierarchy (IH), assumes a fixed, small set of privilege levels (typically fewer than five) defined by rigid role labels (e.g., system > user). This is inadequate for real-world agentic settings, where conflicts can arise across far more sources and contexts. In this work, we propose Many-Tier Instruction Hierarchy (ManyIH), a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels. We introduce ManyIH-Bench, the first benchmark for ManyIH. ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges, comprising 853 agentic tasks (427 coding and 426 instruction-following). ManyIH-Bench composes constraints developed by LLMs and verified by humans to create realistic and difficult test cases spanning 46 real-world agents. Our experiments show that even the current frontier models perform poorly (~40% accuracy) when instruction conflict scales. This work underscores the urgent need for methods that explicitly target fine-grained, scalable instruction conflict resolution in agentic settings.

1 Introduction

ManyIH addresses the fixed- and few-tier bottleneck in current instruction hierarchy by assigning privilege values to instructions and resolving conflicts across arbitrarily many levels. The authors introduce MANYIH-BENCH and find that current frontier models remain brittle as privilege tiers and representations vary.

  • Motivation: Current instruction hierarchy typically uses a fixed, small set of privilege levels, while real-world agents may encounter arbitrarily many tools, skills, and sub-agents.OpenAI’s Model Spec hardcodes five authority levels: root, system, developer, user, and guideline.
  • ManyIH: ManyIH assigns each instruction a privilege value through a dedicated Privilege Prompt Interface and resolves conflicts by comparing ordinal or scalar values.This replaces relative privilege represented through rigid role labels with dynamically assigned values.
  • MANYIH-BENCH: 853 samples span coding and instruction-following tasks, with up to 12 privilege levels per sample and coverage of 46 real-world agents.The coding subset adds conflicting style instructions, while the instruction-following subset adds synthetic privilege conflicts to existing trajectories.
  • Findings: ∼40% accuracy is achieved by frontier models such as GPT 5.4 and Claude Opus 4.6 on ManyIH.The reported result is below 50% accuracy overall.
  • Findings: >8% accuracy drops result from changing only the privilege-information prompt format, while adherence degrades as instruction tiers increase.The sensitivity result is reported for frontier models including GPT 5.4 and Opus 4.6.

2 Preliminary Definitions and Broader Context

The paper defines instructions as the smallest natural-language commands influencing model behavior within an input conversation, and frames Instruction Hierarchy as a rule for resolving conflicts by privilege. Existing IH assigns privilege through a few fixed message-role labels, treating all instructions within each role as equally trusted, which creates a bottleneck in agentic settings.

  • Definitions: An input is the entire conversation history, a message is one conversational turn, and an instruction is the smallest natural-language command that influences model behavior.A single message can contain multiple instructions.
  • Background on Instruction Hierarchy: Instruction Hierarchy resolves conflicts by having higher-privilege, more-trusted instructions override lower-privilege instructions.The paper presents IH as important for LLM safety because many safety challenges can be framed as instruction conflicts.
  • The Fixed- and Few-Tier Bottleneck of Existing IH: Existing IH uses the fixed role ordering system > developer > user > assistant > tool, assigning equal privilege to all instructions sharing a message type.The paper identifies this few-tier design as a fundamental issue because real agentic settings can involve more differentiated trust levels.

3 Design Choices for Many-Tier Instruction Hierarchy

ManyIH replaces fixed role-based privilege tiers with a privilege prompt interface that dynamically encodes arbitrarily many instruction priorities and resolves conflicts by their relative ordering. Its design decouples privilege from message roles, enabling different privilege levels within any token sequence or single message.

  • Privilege Prompt Interfaces: ManyIH uses a dedicated privilege prompt interface to encode dynamically specified privilege values separately from chat-template message roles.The model must use the privilege value specified in the interface when resolving instruction conflicts.
  • Privilege Prompt Interfaces: Each instruction receives a privilege value, is transformed into a tagged token sequence, and is combined with a meta-instruction specifying conflict-resolution rules.The final input concatenates the meta-instruction with all privilege-encoded instructions.
  • Privilege Prompt Interfaces: ManyIH provides ordinal and scalar interfaces: ordinal privileges use lower-number precedence, while scalar privileges use values where the larger value wins.The scalar interface permits inserting an intermediate privilege value between any two existing privileges.
  • Privilege Specification: ManyIH assumes privilege values are supplied at inference time and that privilege is independent of instruction position.Values may be predetermined by developers and deployers according to each instruction source’s trustworthiness.
  • Advantages over Existing IH: ManyIH supports arbitrary privilege levels dynamically specified at inference time and allows privilege to be defined over any token sequence, including within a single message.This decouples instruction privilege from message granularity.

4 MANYIH-BENCH Benchmark

MANYIH-BENCH evaluates multi-tier instruction conflict resolution through non-adversarial, granular tasks spanning coding and instruction-following settings. It uses randomly assigned privileges and a strict all-constraints evaluation criterion.

  • Benchmark Design: MANYIH-BENCH uses straightforward, non-adversarial conflicts to isolate multi-tier resolution from robustness against sophisticated attacks.This separates ManyIH evaluation from complementary attack-robustness capabilities.
  • Task Setup and Statistics: 853 samples comprise coding and instruction-following subsets using the ordinal privilege prompt interface by default.The coding subset contains 427 MBPP-based Python programming samples with conflicting style instructions.
  • Task Setup and Statistics: The coding subset enables controlled difficulty and programmatic evaluation, while instruction-following tests naturalistic, variable-length ManyIH settings.Coding instructions and verification functions are author-curated; instruction-following conflicts are LLM-generated and human-verified.
  • Evaluation: A sample passes only when every active instruction is satisfied, with all coding unit tests also passing.Privileges are randomly sampled, conflicts are resolved using the PPI, and active instructions are determined programmatically.

5 MANYIH-BENCH Benchmark Construction

MANYIH-BENCH constructs coding and instruction-following subsets through a shared pipeline that curates conflicting instructions, composes them into instances, and assigns privilege levels to resolve conflicts. The subsets instantiate this process using code-style constraints and privilege-annotated instructions inserted into agentic prompts.

  • Shared construction pipeline: Both subsets share instruction-bank curation, conflict-aware composition, and privilege assignment that programmatically determines winning instructions.Privilege assignment compares conflicting instructions using their assigned privilege values.
  • Coding Subset: The coding subset pairs MBPP problems with conflicting PEP 8-inspired style instructions curated into 12 groups covering conventions such as indentation, naming, quotes, and operator spacing.Each group contains 2–5 style instructions; the indentation group includes 2-space, 4-space, and tab alternatives.
  • IF Subset: The instruction-following subset augments AgentIF’s multi-turn prompts across 46 agents by inserting privilege-annotated conflicts around granular source instructions.The pipeline first identifies instructions with source spans and classifies whether meaningful opposing instructions can be constructed.
  • IF Subset: Generated instructions undergo LLM and human evaluation, while conflict-group instructions receive distinct privilege levels from 1 to 99 and are randomly shuffled before insertion near anchor spans.The final evaluation set separates the resulting cases after generated instructions are inserted adjacent to their anchors.
  • Construction challenge: The pipeline uses LLMs for individual-instruction generation and verification, but resolving combinations across multiple privilege levels remains combinatorially harder than single-step generation.The paper attributes this multi-level conflict-resolution difficulty to the challenge observed in the benchmark experiments.

6 Experiment and Analysis

Experiments on MANYIH-BENCH show that frontier models struggle as instruction-hierarchy tiers increase, with accuracy declining across nearly all model transitions. Analyses further indicate that reasoning effort, privilege representation, and style compliance materially affect performance, but do not eliminate the ManyIH challenge.

  • Experimental Setup: Ten frontier proprietary and open-source models were evaluated on MANYIH-BENCH with 40K maximum generation length, temperature 0, and high reasoning effort when available.The evaluation included Gemini 3.1 Pro, GPT-5.4, Claude Opus 4.6 and Sonnet 4.6, Grok 4.20 Beta, Kimi K2.5, and four Qwen 3.5 sizes.
  • Scaling Instruction-Hierarchy Tiers: 11 of 12 model–transition pairs show strictly decreasing accuracy as instruction-hierarchy tiers increase, with drops ranging from 6.8% to 24.1%.The smallest drop was for Qwen3.5-9B and the largest was for Sonnet 4.6 between the easiest and hardest configurations.
  • Reasoning Effort: GPT-5.4 accuracy improves monotonically from 15.5% at none reasoning effort to 60.9% at high, while Claude models dip at low effort before recovering.Claude accuracy drops from none to low effort for Sonnet 4.6, 25.3% to 14.1%, and Opus 4.6, 27.9% to 18.7%.
  • Reasoning Behavior: GPT-5.4, the best-performing model, saturates around 60%, indicating that increasing reasoning effort alone is unlikely to resolve ManyIH.Reasoning verbosity also varies substantially, from approximately 1K median tokens for Claude Opus 4.6, Sonnet 4.6, and GPT-5.4 to approximately 7K for Qwen 3.5-397B.
  • Privilege Representation: Scalar privilege values are perturbed by δ ∼Uniform(−3, +3) while preserving strict privilege ordering, testing sensitivity to numerical values beyond relative priority.The benchmark reports non-trivial sensitivity to these perturbations, including changed binary pass/fail outcomes for some samples.
  • Coding Accuracy Breakdown: Style compliance is the primary bottleneck for overall coding accuracy because privilege-based style conflicts require ManyIH reasoning.The performance breakdown separates functional correctness, Acctest, from style compliance, Accstyle.

7 Related Work

Prior work addresses instruction hierarchy through curated training data, architectural role representations, and reasoning-based privilege resolution. Related benchmarks evaluate instruction following, constraint adherence, conflicting instructions, and code-generation adherence, but ConInstruct lacks privilege-based resolution.

  • Instruction Hierarchy: Instruction hierarchy research spans curated training data, learned segment embeddings, and reasoning-based fine-tuning for privilege resolution.Wallace et al. (2024) formalize instruction-hierarchy rules; Wu et al. (2025) distinguish instruction roles architecturally; Zheng et al. (2026) treat resolution as reasoning.
  • Benchmarking Instruction Following and Constraint Adherence: IFEval evaluates instruction following with programmatically verifiable constraint templates, while IFBench extends evaluation to out-of-domain constraints.These benchmarks focus on constraint-based instruction-following evaluation.
  • Benchmarking Instruction Following and Constraint Adherence: ConInstruct studies conflicting instructions without privilege-based resolution, whereas CodeIF benchmarks instruction adherence across diverse code-generation tasks.The cited benchmarks cover conflict handling and code-generation adherence through distinct evaluation settings.

8 Conclusion

ManyIH extends instruction hierarchy from fixed role-based tiers to arbitrarily many privilege levels specified dynamically at inference time. The paper introduces two privilege prompt interfaces and ManyIH-Bench, an 853-sample benchmark covering coding and instruction-following tasks with up to 12 tiers.

  • ManyIH extends instruction hierarchy beyond fixed role-based tiers to arbitrarily many privilege levels specified dynamically at inference time.
  • The paper proposes two privilege prompt interfaces and constructs ManyIH-Bench with 853 samples spanning agentic coding and instruction-following tasks.
  • Up to 12 privilege tiers appear per ManyIH-Bench sample, while frontier models achieve at most ∼40% accuracy in the reported experiments.

Ethics Statement

The proposed privilege prompt interface supports dynamically specified privilege values for resolving instruction conflicts, but it carries a dual-use risk because attackers could assign high privilege to malicious instructions.

  • Ethics Statement: The privilege prompt interface lets models resolve instruction conflicts using dynamically specified privilege values for trusted deployments.The authors give safety-critical system instructions as an example of legitimate use.
  • Ethics Statement: The mechanism creates a dual-use risk: adversaries could tag malicious instructions with high privilege values to manipulate model behavior.The benchmark is described as making an assumption, but the supplied passage truncates before specifying it.

B Human Validation of LLM-generated Constraints … E Coding Subset Variant Statistics

The paper validates LLM-generated constraint checks through manual review and specifies ordinal and scalar privilege-resolution prompts for instruction-following and coding tasks. It also introduces benchmark and coding-variant statistics, though the supplied passages provide only their table descriptions rather than numerical entries.

  • B Human Validation of LLM-generated Constraints: 81 of 100 randomly selected constraint checks were faithful, while 11 were unclear and 8 were incorrect.The reviewed tuples paired a constraint and context with either a prompt or code check.
  • B Human Validation of LLM-generated Constraints: The most common validation failure involved partially ambiguous conditional instructions, whose synthesized prompts and programs often required the conditioned event to occur.The passage gives “always deliver tool calls using XML tags” as an example.
  • B Human Validation of LLM-generated Constraints: Other failures included mismatched check modalities, incomplete constraint checks, and extreme misinterpretations of the required behavior.The examples include using code instead of a semantic prompt check, checking only morning meetings, and requiring the agent to state compliance.
  • B Human Validation of LLM-generated Constraints: Heuristic-based checks were marked incorrect because adversaries could trivially exploit them, although their downstream effects were expected to affect only a small subset of cases.
  • C Prompts Used in Privilege Prompt Interface: The ordinal interface resolves conflicts by giving priority to the lower privilege number, with later instructions winning when privilege numbers are equal.The prompt uses [[Privilege N]]...[[/Privilege]] markers and instructs models to follow as many instructions as possible.
  • C Prompts Used in Privilege Prompt Interface: The scalar interface resolves conflicts by giving priority to the higher z value, with later instructions winning when scalar privilege levels are equal.The prompt uses [[z=N]]...[[/z]] markers and instructs models to follow as many instructions as possible.
  • D MANYIH-BENCH Statistics: MANYIH-BENCH statistics are reported in Table 3, which measures input length in whitespace-delimited words across all model-input messages.The table description states that coding difficulty parameters are fixed by design, whereas instruction-following complexity varies naturally across real agentic scenarios.
  • E Coding Subset Variant Statistics: Table 4 compares three coding-subset configurations in the IH tier-scaling experiment, increasing instructions, privilege levels, and conflicts while holding style groups and winning styles constant.The supplied passage describes the table but does not provide its numerical entries.

F Example Datapoints … G Coding Subset Style Constraints Details

The examples demonstrate ManyIH conflict resolution in coding and instruction-following tasks using ordinal or scalar privilege encodings. The coding subset defines 12 style groups with checker-verifiable constraints, while instruction-following examples apply winning directives across realistic agent prompts.

  • F.1 Coding Subset Examples: The coding example asks for a function computing the n-th hexagonal number while combining 12 style instructions across four groups with 9 pairwise conflicts.The style groups concern quotes, type hints, license, and variable-name length.
  • F.1.1 Ordinal format: Ordinal privilege encoding makes lower numbers dominant, so Privilege 1 overrides Privilege 2 and subsequent levels.Instructions are marked with [[ Privilege N ]]...[[/ Privilege ]] tags.
  • F.1.1 Ordinal format: In the ordinal example, the winning constraints select double quotes and triple-double-quoted docstrings, full type hints, an MIT license, and variable names of at least five characters.The winners are reported by style group, with lower privilege numbers prevailing in conflicts.
  • F.1.2 Scalar format: Scalar privilege encoding makes higher z dominant, with later prompt position resolving conflicts between instructions at the same privilege level.The scalar interface uses [[z=N]]...[[/z]] markers and produces the same expected output as the ordinal interface.
  • F.2 IF Subset Example: The therapist example contains 2 conflict groups, 4 active constraints, and 3 suppressed constraints.Privilege 1 wins both the conflict over sharing personal experiences and the conflict over offering encouragement.
  • F.2 IF Subset Example: The therapist agent must empathize, use gentle humor, offer encouragement, and avoid sharing personal experiences, while untagged empathy and humor instructions remain active.The winning constraints suppress breakup, workplace-only, and non-optimistic-response alternatives.
  • F.3 IF Subset Example: Coding Agent: The Codex CLI example pits solution completeness and code/tool visibility directives against one another, with Privilege 1 requiring TODOs and hiding all code and tool details.The prompt asks the agent to list class names in Python files under the current project’s src/models directory.
  • G Coding Subset Style Constraints Details: The coding subset contains 12 style groups and 41 style constraints, with same-group conflicts, cross-group compatibility, and deterministic AST or token-based verification.The constraints are inspired by PEP 8 and prior work, and stricter compatible constraints can satisfy lenient ones.

H Details on IF Benchmark Creation … H.5 Step 3b (continued): Re-generation After Verification Failure

The instruction-following conflict pipeline combines LLM-based extraction, classification, generation, and verification with programmatic downstream processing. Failed generated constraints receive one feedback-based replacement attempt designed to preserve localized, feasible conflicts.

  • H Details on IF Benchmark Creation: The pipeline uses Claude Sonnet 4.6 for span extraction, conflictability classification, conflict generation, and conflict verification, while Steps 4–7 are fully programmatic.The programmatic steps assign privileges, perform structural verification, prepare evaluations, and convert privileges to ordinals.
  • H.1 Step 1: Source Span Extraction: Step 1 identifies every verbatim source span from which each constraint originated, including paraphrased constraints and non-contiguous spans.Multiple spans are separated with |||, and untraceable constraints are marked NOT_FOUND.
  • H.2 Step 2: Conflictability Classification: Step 2 labels each constraint YES or NO according to whether a meaningful opposing instruction can be constructed.The classifier rejects vague, domain-knowledge, task-description, nonsensical, or trivial conflicts and provides a one-sentence rationale.
  • H.3 Step 3: Conflict Generation: Step 3 generates 1–4 mutually exclusive constraints per selected anchor, each directly conflicting with the anchor and accompanied by a deterministic evaluation rule.Generated constraints must avoid conflicts outside their group and must remain clear, actionable, and feasible for the overall instruction set.
  • H.3 Step 3: Conflict Generation: For code evaluations, generated constraints include a Python check_following(response: str) -> bool function that returns True when the response satisfies the constraint.For LLM evaluations, the generated prompt includes a {response} placeholder and requires a direct YES/NO answer.
  • H.4 Step 3b: Conflict Verification: Step 3b verifies whether each generated constraint causes cross-group conflict or makes the overall instruction set nearly impossible to follow.Only genuine violations of outside constraints or infeasibility are flagged; minor thematic overlap is acceptable.
  • H.4 Step 3b: Conflict Verification: Verification returns PASS, CROSS_GROUP_CONFLICT, or INFEASIBLE for each generated constraint, with a rationale and, for cross-group failures, the conflicting constraint identifier.The reviewer evaluates each generated constraint against the full original constraint set and its conflict groups.
  • H.5 Step 3b (continued): Re-generation After Verification Failure: After a verification failure, the pipeline makes one re-generation attempt using the failure reason and other constraints as feedback.The replacement must conflict with the anchor, avoid all listed other constraints, preserve feasibility, and be returned as a one-element JSON array.
Loading 2604.09443v3…