PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalizing Verifiable Instruction Following

Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, Hannaneh Hajishirzi

arXiv:2507.02833v3cs.CL

TL;DR

Most models overfit to familiar verifiable instruction constraints and generalize poorly to unseen constraints. The paper introduces IFBENCH and IF-RLVR, finding that reinforcement learning with verifiable rewards improves precise instruction following across model families.

  • Problem

    Most models overfit to a small set of benchmarked verifiable constraints and struggle to generalize precise instruction following to unseen constraints.

  • Method

    The paper introduces IFBENCH with 58 new verifiable constraints, IFTRAIN with 29 training constraints, and IF-RLVR training methods using verifiable rewards.

  • Results

    IF-RLVR raises TÜLU-3-8B from 28.9 to 45.9 on IFBENCH and improves precise instruction following across OLMo, Qwen 2.5, and Llama 3.1 model families.

  • Takeaways & Limitations

    IFBENCH remains challenging despite targeted training, while varied training constraints and IF-RLVR improve generalization of precise instruction following.

  • Takeaways & Limitations

    The work focuses exclusively on verifiable constraints, which excludes many real-world constraints without easily verifiable ground truth and can make some constraints seem contrived.

Abstract

from arXiv · show

A crucial factor for successful human and AI interaction is the ability of language models or chatbots to follow human instructions precisely. A common feature of instructions are output constraints like ``only answer with yes or no" or ``mention the word `abrakadabra' at least 3 times" that the user adds to craft a more useful answer. Even today's strongest models struggle with fulfilling such constraints. We find that most models strongly overfit on a small set of verifiable constraints from the benchmarks that test these abilities, a skill called precise instruction following, and are not able to generalize well to unseen output constraints. We introduce a new benchmark, IFBench, to evaluate precise instruction following generalization on 58 new, diverse, and challenging verifiable out-of-domain constraints. In addition, we perform an extensive analysis of how and on what data models can be trained to improve precise instruction following generalization. Specifically, we carefully design constraint verification modules and show that reinforcement learning with verifiable rewards (RLVR) significantly improves instruction following. In addition to IFBench, we release 29 additional new hand-annotated training constraints and verification functions, RLVR training prompts, and code.

1 Introduction

The paper argues that precise instruction following remains difficult because models overfit to familiar verifiable constraints and generalize poorly to unseen ones. It introduces IFBENCH and RLVR-based training methods to measure and improve this generalization.

  • Most models overfit IFEval’s 25 automatically verifiable constraint templates, despite leading models exceeding 80% at as few as 2B parameters.
  • The evaluation focuses on constraints that can be verified with software, although many real-world user constraints lack easily verifiable ground truth.
  • IFBENCH introduces 58 diverse, challenging, and unseen verifiable constraints spanning counting, formatting, manipulation, and copying.Leading models score below 50% on the benchmark.
  • The authors create 29 hand-curated training constraints and verification functions, called IFTRAIN, to improve precise instruction-following generalization.The constraints target useful, challenging, and real-world-inspired capabilities.
  • RLVR training interleaves multiple constraints or mixes verifiable and preference rewards, improving TÜLU-3-8B from 82.4 to 92.2 on IFEval and from 28.9 to 45.9 on IFBENCH.The approach also raises Qwen-2.5-7B to 87.8 on IFEval and 54.7 on IFBENCH.

2 IFBENCH & IFTRAIN: Measuring and Training Precise IF

IFBENCH evaluates precise instruction-following generalization using 58 new verifiable constraints, while IFTRAIN supplies 29 new training constraints and verification functions. The benchmark exposes overfitting to existing constraints and supports contamination-resistant evaluation.

  • IFBENCH: Models score well on IFEval but much lower on IFBENCH, indicating poor generalization to unseen constraints.The two benchmarks use different constraints despite the same task and evaluation setup.
  • IFBENCH: IFBENCH introduces 58 new verifiable constraints spanning counting, formatting, manipulation, and copying skills.Its constraints are paired with Python verification functions for reproducible evaluation and training tools.
  • IFBENCH: IFBENCH combines unseen constraints with unseen WildChat prompts to reduce accidental train-test contamination.Every instance undergoes human annotation for prompt-constraint compatibility.
  • IFTRAIN: IFTRAIN adds 29 unseen verifiable constraints and corresponding verification functions, more than doubling the existing training constraint types.The constraints target foundational skills such as copying spans and copying while editing input.

3 IF-RLVR

IF-RLVR trains models for precise instruction following with verifiable rewards, using diverse constraint data and GRPO-based outcome supervision. The approach improves performance across model families, with broader multi-constraint training supporting stronger generalization.

  • Training approach: Diverse training prompts combine public SFT instructions with constraints from IFEval or a separate unseen training taxonomy.Prompts contain at least one and up to n constraints.
  • Training approach: IF-RLVR trains policies with GRPO and outcome supervision, scoring each output according to whether its constraints are fulfilled.For multiple constraints, instance rewards combine verifiable rewards with configurable multipliers and weights.
  • Results: IF-RLVR-trained models outperform most current state-of-the-art models and work across OLMo, Qwen 2.5, and Llama 3.1 families.The evaluation includes both in-domain IFEval and out-of-domain IFBENCH settings.

4 IF-RLVR Experiments

The experiments show that broader and more varied constraint training improves precise instruction-following generalization, while IF-RLVR generally outperforms DPO and can improve both base and instruction-tuned models.

  • 4.1 Training on Multiple Constraints: Training on more than one constraint per instance improves both in-domain IFEval and out-of-domain IFBENCH performance.Training on up to five or six constraints can outperform training on only up to three, even though the benchmarks contain fewer constraints per instruction.
  • 4.2 Training Data Composition: Larger and more varied training sets improve generalization, while adding IFEval constraints chiefly increases in-domain IFEval performance.The full combination of IFTRAIN and IFEval constraints produces the highest IFEval performance, whereas IFBENCH is less affected by the number of IFEval constraints.
  • 4.4 Removing Constraint Categories: Removing LENGTH or KEYWORDS constraints harms IFEval most, whereas removing CHANGE CASES or DETECTABLE FORMAT barely changes performance, reaching 89.65 accuracy.This leave-one-category-out experiment evaluates how training on constraint categories transfers to unseen categories.
  • 4.5 Teaching the Basic Units of Precise IF: IF-RLVR improves targeted skills such as counting, keyword insertion, formatting, and length constraints, while many IFEval categories exceed 90 performance.IFBENCH words and sentence categories still leave room for improvement after training.
  • 4.5 Teaching the Basic Units of Precise IF: Compared with other post-training approaches, IF-RLVR achieves the best in-domain and out-of-domain performance on both IFEval and IFBENCH.The gains slightly harm AlpacaEval 2 while remaining comparable on GSM8K, MMLU, and BBH.
  • 4.6 DPO: GRPO with verifiable rewards consistently outperforms DPO on IFEval and IFBENCH when using the same prompts and starting model.Starting from a model that has undergone both SFT and DPO produces higher final instruction-following performance.
  • 4.7 RLVR for IF from Base: IF-RLVR from base models achieves nearly the same IFEval performance as training from instruction-tuned policies, with reasoning improving out-of-domain IFBENCH generalization.The reported conclusion is that IF-RLVR with reasoning improves precise instruction-following generalization.

5 Reward Hacking and the Instruction Hierarchy

The paper examines a trade-off between following challenging output constraints and producing generally high-quality responses when constraints are removed. RLVR improves verifiable constraint adherence but can make models prioritize constraints over the main task.

  • Trade-offs between response quality and instruction following: IFBENCH accuracy is compared with GPT-4.1 LLM-as-a-judge ratings of completions evaluated without their constraints.The comparison uses both IFEval and IFBENCH prompts and contrasts a base policy with an IF-RLVR-trained model.
  • Trade-offs between response quality and instruction following: 7 out of 10 versus 6.4: base-policy completions receive higher average judge scores than IF-RLVR completions when constraints are removed.The base policy scores 7, while the IF-RLVR model scores 6.4 on the judge evaluation.
  • Reward Hacking and the Instruction Hierarchy: IF-RLVR models achieve higher verifiable accuracy but tend to prioritize constraints over answering the full instruction.The paper describes this behavior as over-optimization and proposes combining verifiable rewards with a general reward-model signal.

6 Related Work

Prior work evaluates instruction following through verifiable constraints, increasing constraint complexity, decomposed atomic constraints, and generalization-focused benchmark construction. These approaches differ in constraint type and evaluation method.

  • Precise Instruction Following: Instruction-following remains difficult because reasoning abilities can decline under formatting constraints and fine-grained constrained generation remains challenging.Earlier work improved instruction following through larger instruction fine-tuning, activation steering, and DPO training.
  • Generalization and Benchmark Contamination: Unseen test sets are used in prior work to investigate generalization and detect overfitting, including perturbed GSM8K-Symbolic and analyses of AIME’24.These studies motivate evaluating performance beyond small or familiar benchmarks.
  • Existing Instruction-Following Benchmarks: IFEval evaluates 25 verifiable output-format constraints, whereas FollowBench increases constraint counts and uses LLM judges for non-verifiable situation, style, and format constraints.InFoBench instead decomposes instructions into atomic constraints and rates each one.

7 Conclusion and Limitations

The paper concludes that precise instruction following remains difficult despite targeted training efforts, while identifying important limitations in its focus on verifiable constraints. It calls for broader constraint types in future work.

  • IFBENCH remains challenging despite targeted training efforts, highlighting persistent difficulty in generalizing precise constraint following.
  • The study exclusively focuses on verifiable constraints, excluding many real-world constraints without easily verifiable ground truth.This focus may also make some included constraints seem unnatural or contrived.
  • Future work should explore reinforcement-learning training for a wider variety of constraints, including some that are not necessarily verifiable.

A Out-of-Distribution Test Constraints

The out-of-distribution test suite contains diverse verifiable constraints spanning formatting, counting, ratios, sentence structure, word properties, copying, punctuation, and custom transformations. It also includes structured generation and knowledge-based tasks with precise output requirements.

  • Counting and composition: Counting constraints regulate conjunctions, numbers, person names, pronouns, punctuation, unique words, word counts, Japanese words, lowercase words, letters, and sentence composition.
  • Ratios and sentences: Ratio and sentence constraints specify trigram overlap, sentence-type balance, declarative-to-interrogative ratios, character equality, stop-word proportions, alliteration growth, and incremental sentence lengths.
  • Word properties: Word-level constraints govern alphabet cycling, consonant clusters, sentence-boundary word links, initial-letter repetition, syllable alternation, palindromes, prime lengths, vowels, and repetition limits.
  • Custom tasks: Custom constraints include reversed answers, CSV generation, date formatting, sorted European capitals, progressively longer art-history questions, multiples of seven, reverse-alphabetical countries, and alphabetically sequenced stories.
  • Copying and positional constraints: The suite also tests copying, keyword placement or exclusion, exact letter counts, paragraph counts, word positions, punctuation avoidance, and response-start or response-end requirements.

D.1 Length Analysis

IFBENCH prompts are short in single-turn settings and longer in multi-turn conversations, while frontier models generate substantially longer responses than IF-RLVR-trained models. The benchmark also uses more complex constraints than IFEval.

  • Prompt and response lengths: Single-turn IFBENCH prompts average 76 tokens, while multi-turn conversations average 408 input tokens.The multi-turn length includes the prompt, response, and subsequent prompt.
  • Prompt and response lengths: Frontier models generate responses averaging 2214 tokens, compared with 210 tokens for IF-RLVR-trained models.
  • Constraint complexity: IFBENCH extends IFEval with more challenging mathematical, algorithmic, nested linguistic, and adversarial constraints.Examples include trigram overlap percentages, alphabet cycling, palindromes, and alliteration progression.

E Mitigating Reward Hacking

Verifiable-reward training can over-prioritize output constraints over the full instruction, so the paper combines verifiable rewards with a general preference signal. This combined approach slightly lowers instruction-following scores relative to ground-truth-only training while improving AlpacaEval 2.

  • Reward hacking: RLVR can cause models to over-prioritize output constraints instead of answering the full instruction.The paper describes this behavior as over-optimization or reward hacking.
  • Combined rewards: The proposed mitigation combines verifiable reward Vi with a general reward-model score Si when Vi > 0.The verifiable reward checks constraint adherence, while the general reward model evaluates whether the response answers the prompt.
  • Combined rewards: The final reward adds 1 when Vi > 0 and Si > α, subtracts 0.5 when Vi > 0 and Si ≤ α, and otherwise retains Vi.
  • Implementation: The implementation uses Llama-3.1-Tulu-3-8B-RM, α = 75, an effective batch size of 512, and 8 samples per prompt.
  • Results: After 1100 steps, the combined-reward model scores 86.1 on IFEval, 30 on IFBENCH, and 31.6 on AlpacaEval 2.It scores slightly lower on instruction following than ground-truth-only training while remaining more than 5 points above the base policy.

G Evaluation Details

Evaluation generally uses deterministic generation settings, with model-specific exceptions and token limits. The training setup includes a chat template for IF-RLVR training from a base model.

  • Generation settings: Evaluation generally uses temperature 0, with maximum generated tokens adjusted by model type.Thinking models receive more generation tokens, and outputs are processed to extract answers without reasoning chains.
  • Model-specific settings: Deepseek R1 evaluation uses temperature 0.6 and top p 0.95, following the model’s recommended settings.
  • Model-specific settings: o3 evaluation uses the required temperature of 1.
  • Training template: The IF-RLVR training setup uses a chat template built from a base model.
Loading 2507.02833v3…