PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Up the Instruction Ladder for Controllable Language Models

Zishuo Zheng, Vidhisha Balachandran, Chan Young Park, Faeze Brahman, Sachin Kumar

arXiv:2511.04694v5cs.CLcs.AI

TL;DR

LLMs must reconcile competing instructions to remain reliable and controllable, yet existing models often fail when system and user directives conflict. This paper reframes prioritization as explicit reasoning, trains it with RLVR on VerIH, and reports consistent gains across instruction-following and hierarchy benchmarks, including ∼20% absolute conflict-setting improvements and up to 20% absolute ASR reductions on safety evaluations.

  • Problem

    LLMs often fail to consistently prioritize conflicting system and user instructions, creating reliability and safety challenges.

  • Method

    The method uses RLVR on VerIH to train models to reason about instruction relationships and priorities before executing user requests.

  • Results

    The method consistently improves instruction-following and hierarchy benchmarks, achieves ∼20% absolute gains in conflict settings, and reduces attack success rates by up to 20%.

  • Takeaways & Limitations

    Updating higher-priority system directives can steer behavior beyond the training distribution without retraining for each new instruction.

  • Takeaways & Limitations

    The evidence is limited to two pretrained reasoning-model families, 4B–35B sizes, 7,200 training samples, and a fixed GRPO schema, with broader benchmarks and tasks still needed.

Abstract

from arXiv · show

As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources within a single prompt context. Enforcing an instruction hierarchy, where higher-level directives override lower-priority requests, is critical to the reliability and control of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. The model must first "think" about the relationship between a given user prompt and higher-priority instructions before generating a response. To enable this capability, we construct VerIH, a training dataset of constraint-following tasks with verifiable answers, comprising aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our method leads to consistent improvements across multiple model families on both instruction following and instruction hierarchy benchmarks, achieving ~20% absolute improvement in conflict setups. Our method also leads to improved alignment to safety-critical scenarios beyond the training distribution, exhibiting increased robustness against jailbreak and prompt injection, reducing absolute attack success rates by up to 20%. Our results establish reasoning over instruction hierarchies as a practical mechanism for improving AI reliability, where targeted updates to system prompts produce predictable, controllable, and robust changes in model behavior.

1 Introduction

LLMs often struggle to prioritize conflicting system and user instructions, undermining reliability and safety. The paper reframes instruction prioritization as explicit reasoning, introduces VerIH, and reports consistent gains on instruction-following, hierarchy, and safety evaluations.

  • Conflicting directives from system designers, users, and external tools create reliability and safety challenges, including vulnerability to prompt injection and jailbreaks.
  • Existing approaches largely treat instruction prioritization as input–response mapping, while safety-focused reasoning methods do not cover ordinary or harmless conflicts.
  • The proposed method makes models explicitly reason about the task, instruction sources, and precedence before executing a user request.
  • VerIH extends instruction-following data with aligned and conflicting system–user prompts plus verifiable constraints on response format, quantity, and keywords.
  • ∼20% absolute gains under conflict settings occur across instruction-following and hierarchy benchmarks without losing general capabilities.
  • Up to a 20% absolute reduction in attack success rate appears on safety and jailbreaking benchmarks when higher-priority safety directives are evaluated out of distribution.

2 Reasoning for instruction hierarchy

The paper defines instruction hierarchy as priority-based conflict resolution and trains reasoning-enabled models to resolve system–user conflicts before answering. Its setup focuses on two hierarchy levels and uses RLVR with explicit reasoning and verifiable rewards.

  • 2 Reasoning for instruction hierarchy: Higher-level instructions take precedence over lower-level directives, so conflicting lower-priority requests are rejected or overridden before task execution.
  • 2 Reasoning for instruction hierarchy: The paper transfers existing general reasoning ability to instruction prioritization through reinforcement learning with variable reward (RLVR).
  • 2.1 Problem setup: The hierarchy can include system prompts, user prompts, model outputs, and tool outputs, but experiments simplify it to system prompts S and user prompts U.
  • 2.1 Problem setup: Aligned prompt pairs contain no conflict, whereas conflicting pairs require prioritizing S, rejecting conflicting parts of U, and answering its non-conflicting parts helpfully.
  • 2.2 RLVR training: RLVR scores each prediction with a reward function, using the resulting score as supervision for hierarchy reasoning.
  • 2.2 RLVR training: During training, a SysHint asks the model to reason about system–user relationships before producing an answer represented as CoT plus the final response.

3 VerIH: a dataset with verifiable answers for instruction hierarchy

VerIH adapts instruction-following data into a verifiable dataset containing both aligned and intentionally conflicting system–user prompts. Its construction uses automated rewriting, accepts some noise, and checks that conflicts and verifiers remain usable.

  • VerIH modifies RLVR-IFEval so each original aligned example is paired with system constraints and a user task, while some user prompts are rewritten to conflict with those constraints.
  • 7,192 samples span 24 categories, with half rewritten into conflicting cases and half retained as aligned cases.
  • The dataset construction tolerates noise from imperfect rewriting while still producing consistent and significant benchmark gains.
  • In the vast majority of rewritten samples, conflicts are successfully introduced and the original verification function remains effective.
  • Claude-4-Sonnet generates the conflicting rewrites, while OpenAI GPT variants produced less accurate prompts in qualitative experiments.
  • The dataset examples illustrate system constraints requiring formatting behavior that conflicts with user requests for ordinary concise answers.

4 Experiments

Experiments test instruction following, hierarchy resolution, general reasoning, and safety across multiple model families and training baselines. The evaluation also examines whether higher-priority safety instructions generalize beyond the training distribution.

  • The study evaluates instruction following with IFEval and IFBench, hierarchy with IHEval, general reasoning with MMLU and MATH-500, and safety with Harmbench, WildJailbreak, and TensorTrust.
  • Models are compared across prompting configurations, including no explicit reasoning, chain-of-thought reasoning, and SysHint-based hierarchy reasoning.
  • Supervised fine-tuning baselines distill teacher-generated reasoning traces into filtered and unfiltered datasets, with full-parameter and LoRA settings, alongside a random-reward RLVR baseline.
  • Evaluations use consistent non-safety configurations and a default system prompt that instructs models to follow user prompts faithfully when no system prompt is provided.
  • Safety generalization is tested by adding GuardRules as higher-priority system instructions, although no safety-related samples appear in training.
  • GuardRules prohibit stereotyping, offensive language, dangerous advice, and copyrighted-content generation while preserving helpfulness and response-length guidance.
  • The model suite spans Qwen3 sizes from 4B to 35B, a Mixture-of-Experts model, and Phi-4-mini-reasoning, with finetuning details deferred to Appendix C.

5 Results

Across model families, VerIH improves instruction-following and instruction-hierarchy performance while preserving general reasoning, and it generalizes to safety benchmarks without safety-specific training.

  • +16.42, +7.17, and +7.47 points on IFBench and +22.87, +17.00, and +18.21 points on IHEval-conflict are reported for Qwen3 4B, 8B, and 14B.
  • VerIH improves most instruction-following and instruction-hierarchy benchmarks while maintaining or slightly improving general reasoning performance across models.Table 1 summarizes results across instruction-following, instruction-hierarchy, and general benchmarks.
  • RLVR with VerIH achieves the best overall trade-off, improving instruction-hierarchy performance while preserving general reasoning and reducing attack success rates versus SFT and LoRA.Full SFT substantially degrades general reasoning ability, whereas LoRA reduces some degradation but does not match base-model general performance.
  • Safety performance improves consistently even without safety-related training samples, supporting generalization from instruction prioritization to jailbreak and prompt-injection benchmarks.Table 3 identifies safety as a special case of instruction conflict and reports consistent improvements across these benchmarks.

6 Analysis

The analysis shows that VerIH’s gains depend on training with conflicting prompts and explicit hierarchy reasoning, generalizing to realistic, multi-turn, and safety-related settings.

  • 6.1 Ablation Study: +VerIH achieves the strongest performance across all benchmarks and models, while ablations consistently degrade performance.Aligned-only training remains competitive on aligned benchmarks but drops 10–25 percentage points on conflicting benchmarks.
  • 6.2 Real-world scenarios: 89.70% compliance is achieved by Qwen3-8B on Model Spec Evals with VerIH and policy prompts.This performance is comparable to GPT-5 Thinking variants and exceeds the CoT+SysHint+Policy comparison.
  • 6.3 Reasoning traces analysis: 77.88% and 91.53% are the IH explicit reasoning ratios after VerIH training on IHEval:aligned and IHEval:conflict, respectively.The corresponding pre-VerIH ratios are 65.43% and 68.06%.
  • 6.4 Extending to multi-turn conversations: ∼20 points improvement is achieved on the IHEval:conflict setup, including gains that generalize to multi-turn cases.VerIH uses simple single-turn conflicts, yet its gains extend beyond the training distribution and synthetic setup.
  • 6.5 Test-time scaling: Budget forcing does not benefit these released models because Qwen3 and Phi-4-mini-reasoning already generate “wait” tokens to extend reasoning.The evaluation examined test-time compute scaling on IHEval after VerIH training.

7 Related work

Related work improves instruction following through supervised and preference-based training, prompt representations, reinforcement learning, and reasoning, but commonly treats hierarchy resolution or safety separately.

  • 7.1 Instruction following and hierarchy: Instruction following has progressed from human-annotated SFT and RLHF to RLVR, self-improvement, and explicit reasoning methods.These approaches address instruction-following ability but primarily focus on aligned prompts.
  • 7.1 Instruction following and hierarchy: Instruction hierarchy methods distinguish prompt priorities or train compliance with privileged instructions, yet models still struggle when system and user instructions conflict.The paper frames its contribution as combining instruction hierarchy with reasoning through RLVR training.
  • 7.2 Safety reasoning: Safety defenses remain vulnerable to prompt injection and jailbreaks because models lack reliable instruction–data separation and static defenses require retraining for updates.Existing methods rely on internalized safety knowledge and may lack robustness to novel or adversarial scenarios.
  • 7.2 Safety reasoning: The proposed distinction is that VerIH explicitly enforces instruction-hierarchy reasoning as a dynamic and generalizable alternative to safety-specific training.The related-work discussion positions safety as an outcome of resolving competing instructions rather than the sole training target.

8 Conclusion

The conclusion presents instruction hierarchy as a meta-reasoning problem: models prioritize layered human intent before execution, and VerIH training transfers this ability beyond synthetic tasks.

  • 8 Conclusion: The paper reframes instruction hierarchy as meta-reasoning that integrates and prioritizes instructions before execution.RLVR training uses VerIH examples containing aligned and conflicting system–user prompts.
  • 8 Conclusion: Training on a simple constraint-following hierarchy dataset generalizes IH reasoning to downstream safety domains without domain-specific finetuning.The conclusion identifies this out-of-distribution transfer as its most interesting observation.
  • 8 Conclusion: The reported improvements target controllability and robustness of instruction execution, especially under adversarial prompts.Inference-time prioritization is described as helping models resist interfering inputs while adhering to higher-priority instructions.

Limitations

The experiments are limited by benchmark coverage, computational resources, model families, training scale, and the token cost of reasoning before execution.

  • Limitations: The evaluation needs extended benchmarks and more downstream tasks, especially multi-round dialogue and auto-attacking methods.These settings may reveal additional challenges for instruction-hierarchy reasoning.
  • Limitations: Compute constraints limited experiments to 4B–35B models, 7,200 training samples, and GRPO with group_size=4.The study uses two families of pretrained reasoning models and a fixed training schema.
  • Limitations: Reasoning before task execution introduces increased token cost, creating an unresolved effectiveness–efficiency trade-off.The paper identifies reducing chain-of-thought overhead as a direction for future work.

Ethical considerations

The work acknowledges dual-use risk because its training recipe could be adapted to make models ignore higher-priority prompts. It also documents the conflict-generation process, which rewrites user requests against system constraints.

  • Ethical considerations: The method carries dual-use risk because malicious actors could adapt it to train models that violate higher-priority prompts, undermining safety mechanisms or propagating harmful content.The authors emphasize responsible deployment, monitoring, and alignment safeguards.
  • Ethical considerations: The dataset construction uses Claude-4-Sonnet to generate subtle conflicts that may be difficult to detect.The rewriting prompt asks for an explanation of the modification within 20 words.
  • Ethical considerations: VerIH conflict examples are created by rewriting user prompts while preserving the original system constraint and task intent.The supplied examples add a detailed-review request that conflicts with a yes/no/maybe response requirement.

B Dataset Quality Analysis

The analysis examines whether LLM-generated rewriting produces genuine conflicts while preserving compatibility with the original reward functions. It also reports generalization from two-level instruction hierarchies to tool-use settings.

  • Dataset quality analysis: 87.53% of sampled rewritten examples were judged to contain conflicts, which the authors consider acceptable despite possible LLM-based analysis errors.The authors tolerate some rewriting noise because the method still achieves gains across benchmarks.
  • Reward compatibility: An approximately 8% lower pass rate after rewriting is attributed to conflict samples being harder, while remaining compatible with the original reward functions.The comparison uses 500 rewritten conflict samples and 500 aligned samples.
  • Evaluation scope: Training evaluates instruction following, instruction hierarchy, general reasoning, and safety-related tasks, including aligned and conflicting IHEval cases.MMLU and MATH-500 test whether VerIH finetuning degrades general reasoning capabilities.
  • Multi-level instruction hierarchies: The model generalizes to a three-level system–user–tool hierarchy despite training data lacking tool_calls examples, gaining 5% on the conflict tool-use setup.The proposed multi-level design places prompts at different roles and expects two-level training to generalize.

F Full ablation study results

The ablations indicate that explicit reasoning, conflict examples, and SysHint shape instruction-hierarchy behavior, while model traces show both genuine prioritization and superficial compliance failure modes.

  • Full ablation study results: Disabling reasoning during training can produce superficial format compliance: Phi-4-mini-reasoning reaches 45.37% on IFBench but emits meaningless fragments instead of complete sentences.The authors interpret this as overfitting to prompt-format constraints rather than genuine instruction understanding.
  • Reasoning traces: SysHint initially increases explicit instruction-hierarchy reasoning in chain-of-thought traces, and VerIH training amplifies that effect.The analysis uses Claude-4-Sonnet to classify whether traces reason about system–user relationships.
  • Ablation study for SysHint: SysHint removal leaves most benchmarks similar, improves IFEval by 4.44%, but raises TensorTrust attack success rate to 20.15%.The authors suggest SysHint supports generalization to unseen domains and complex hierarchy resolution.
  • Test-time compute: Increasing test-time token budgets produces no significant IHEval improvement, suggesting Qwen3-8B already incorporates test-time scaling in its reasoning traces.Budget forcing therefore adds no observed gain after RLVR training.
  • Reasoning traces: After RLVR, traces explicitly identify conflicts and prioritize system instructions over conflicting user requests.Examples include choosing verb extraction over a user-requested translation and limiting capitalization despite a conflicting user constraint.
Loading 2511.04694v5…