PaperlayerView exact revision on arXiv

Source-linked AI summary

ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modeling

Bosi Wen, Yilin Niu, Xiaoying Ning, Ying Zhang, Hongning Wang, Minlie Huang

arXiv:2609.32189v1cs.CL

TL;DR

Scope-aware constraints govern particular response segments, yet existing methods often neglect scope in data construction and rely on sparse binary rewards. ScopeIF uses a Scope–Target–Range schema, a diverse instruction dataset, and tool-grounded graded rewards. It consistently outperforms existing frameworks, especially on complex scope-aware constraints, and its optimized Qwen3 models rival or surpass strong frontier models.

  • Problem

    Existing methods often neglect constraint scope during data construction and use binary per-constraint rewards, limiting data diversity and producing sparse supervision for complex constraints.

    Source: §1 · p. 2

  • Method

    ScopeIF factorizes constraints into Scope, Target, and Range, builds a diverse scope-aware instruction dataset, and combines tool-grounded verification with graded reward modeling.

    Source: §1 · p. 2, §4.1 · p. 4, §5.2 · p. 6

  • Results

    ScopeIF consistently outperforms existing frameworks, particularly on complex scope-aware constraints, and enables optimized Qwen3-4B and 8B models to rival or surpass Gemini-2.5-Pro and DeepSeek-V3.2.

    Source: §1 · p. 2, §6.2 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    The results support ScopeIF as an effective framework for advancing scope-aware instruction-following.

    Source: §7 · p. 10

    Tool-grounded verification adds computational overhead during reward calculation, though this cost is incurred only during training and does not affect inference efficiency.

    Source: F LIMITATIONS · p. 21

Abstract

from arXiv · show

Precise instruction-following is a fundamental ability of large language models (LLMs), requiring their outputs to strictly satisfy objective constraints in input instructions. In complex application scenarios, these constraints often possess diverse scopes that govern specific response segments rather than the entire output. However, existing optimization methods often neglect constraint scope during data construction and rely on binary per-constraint rewards, yielding limited data diversity and sparse supervision for complex constraints. To this end, we propose ScopeIF, a novel training framework for scope-aware precise instruction-following. We first introduce a unified schema that factorizes objective constraints into three decoupled dimensions: Scope, Target, and Range. Grounded in this schema, we construct ScopeInstruct, a large-scale instruction dataset with diverse scope-aware constraints, and combine tool-grounded verification with graded reward modeling to quantify the violation degree of each constraint, providing dense supervision for policy optimization. Extensive experiments demonstrate that ScopeIF consistently outperforms existing methods, particularly on complex scope-aware constraints, while preserving general capabilities. Notably, it enables optimized Qwen3-4B and 8B models to rival or surpass strong frontier models such as Gemini-2.5-Pro and DeepSeek-V3.2, establishing an effective paradigm for advancing scope-aware instruction-following. Our code and data are available at https://github.com/thu-coai/ScopeIF.

1 INTRODUCTION

Scope-aware constraints govern specific response segments, a capability current models still struggle with despite proficiency on whole-response constraints. ScopeIF combines a factorized constraint schema and graded rewards to address this gap, with reported gains on complex constraints.

1 INTRODUCTION

  • Current models handle simple whole-response constraints better than scope-aware constraints governing particular response segments.Scope-aware constraints require fine-grained control and planning, and are prevalent in complex real-world instructions.Source: §1 · p. 1
  • Existing methods neglect scope during data construction and use binary per-constraint rewards that provide sparse, ineffective supervision for complex constraints.The constraints may cover multiple segments or require intersegment dependencies.Source: §1 · p. 2
  • ScopeIF factorizes objective constraints into three dimensions and uses tool-grounded verification with graded reward modeling for dense RL supervision.The schema supports diverse scope-aware constraint synthesis and violation-degree computation.Source: §1 · p. 2
  • ScopeIF consistently improves scope-aware instruction-following, particularly on complex constraints.The authors report validation across multiple policy models and benchmarks.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates and optimizes precise instruction-following with verifiable constraints, but scope modeling and reward granularity remain limited. ScopeIF targets both limitations through scope-aware synthesis and graded rewards.

2 RELATED WORK

  • Precise instruction-following research spans output length and format constraints, with IFEval, IFBench, and IFHierBench extending evaluation to broader categories and hierarchical or scope-aware constraints.These benchmarks still rely on a narrow set of hand-crafted, code-verifiable constraints, unlike ScopeIF’s automatic diverse synthesis.Source: §2 · p. 3
  • Instruction optimization has progressed from supervised fine-tuning through preference optimization to RL, but common approaches neglect scope modeling and use binary rewards that miss partial compliance.ScopeIF instead combines scope-aware data synthesis with graded rewards to provide denser supervision.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define precise instruction-following as satisfying objective constraints and distinguish constraints that apply to response segments. They describe instruction- and constraint-level accuracy as binary measures that can produce sparse rewards for complex constraints.

3 PRELIMINARIES

  • An instruction is followed when the response satisfies all its objective, deterministic constraints; scope-aware constraints apply to specific response segments.Such segment-specific constraints are prevalent in complex real-world instructions.Source: §3 · p. 3
  • Instruction-Level Accuracy requires every constraint to be satisfied, while Constraint-Level Accuracy measures the percentage of satisfied atomic constraints.Both metrics rely on binary signals and do not measure violation degree, creating sparse rewards for complex constraints.Source: §3 · p. 3

4 DATA CONSTRUCTION

Data Construction represents objective constraints with a shared Scope–Target–Range schema, then recombines dimensions and uses LLM-based crossover to create diverse multi-constraint instructions. Quality control filters generated data before release as SCOPEINSTRUCT.

4.2 INSTRUCTION GENERATION PIPELINE

  • The generation pipeline creates atomic constraints by recombining compatible schema dimensions, then uses LLM-based crossover to assemble instructions with multiple constraints.Crossover targets k values of 2, 4, 6, or 8 constraints per instruction.Source: §4 · p. 4, §4.2 · p. 5

4.1 UNIFIED CONSTRAINT SCHEMA

  • The Scope–Target–Range decomposition supports diverse constraint synthesis through dimension recombination and provides a shared basis for quantifying violation degrees.It addresses prior methods’ indivisible templates and manually designed constraint inventories.Source: §4.1 · p. 4
  • Scope selects governed response segments, Target specifies the measurable object in each segment, and Range sets its permissible bounds.Scope can select multiple segments, such as every sentence or paragraph.Source: §4.1 · p. 4
  • The schema covers scopes from the whole response to nested or condition-selected segments, targets from length and literals to structure and relations, and absolute or relative ranges.The taxonomy includes Global, Traversal, Position, and Condition scopes; six target categories; and six range categories.Source: §4.1 · pp. 4–5

4.2 INSTRUCTION GENERATION PIPELINE

  • Quality control removes instructions with ambiguous intents, conflicts, unreasonable requests, or inaccurate constraint lists, yielding 16,968 training and 1,000 test instructions.Each instruction contains 1 to 12 constraints, with 4.83 on average.Source: §4.2 · p. 5

5 METHODOLOGY

ScopeIF combines tool-grounded verification with graded rewards to provide dense supervision for RL optimization, using GRPO for training.

5.1 TOOL-GROUNDED CONSTRAINT VERIFICATION

  • A judge designs response-tailored tools when needed, and an interpreter executes them before the judge derives granular statistics for each constraint’s targets.This approach addresses the challenge of isolating relevant response segments and counting elements under diverse delimiters.Source: §5.1 · pp. 5–6
  • Verification records pair each target object’s permissible bound with its measured value, enabling a binary constraint-satisfaction indicator and subsequent graded reward computation.Source: §5.1 · p. 6

5.2 GRADED CONSTRAINT REWARD MODELING

  • Each target object receives an exponentially decayed reward based on its normalized deviation from the permissible bound, and object rewards are combined by geometric mean.The normalized violation divides absolute deviation by the object’s characteristic range magnitude.Source: §5.2 · p. 6
  • The final reward combines binary satisfaction with graded rewards for violated constraints, preserving discrete leaps for satisfying more constraints while distinguishing partial compliance.This hierarchy provides dense supervision for failures and is integrated with a format reward for GRPO training.Source: §5.2 · p. 6

6 EXPERIMENTS

ScopeIF improves instruction-following across model scales and frameworks, with its strongest advantages on complex scope-aware constraints. The analysis also reports more diverse training constraints, stronger verification, and denser reward signals.

6.2 MAIN RESULTS

  • Average relative improvements of 60.7% on IFBench and 44.7% on IFHierBench let optimized Qwen3-4B and 8B rival or surpass Gemini-2.5-Pro and DeepSeek-V3.2.ScopeIF also surpasses alternative optimization methods and existing instruction-following frameworks, with a marginal gap on IFEval’s simple constraints and a substantial lead on complex scope-aware benchmarks; general performance degrades negligibly.Source: Table 1 · p. 7, Table 2 · p. 7, §6.2 · p. 8

6.1 EXPERIMENTAL SETUP

  • Experiments compare Qwen3-4B and Qwen3-8B against off-the-shelf models, alternative optimization methods, and instruction-following frameworks across instruction-following and general benchmarks.The instruction-following evaluation includes IFEval, IFBench, IFHierBench, and a test set; the test set uses CSR and ISR metrics.Source: §6.1 · pp. 6–7

6.3 ANALYSIS

  • Base-model performance falls from Global constraints through single-layer selectors to recursively composed Nested scopes, while ScopeIF’s advantage over binary-reward baselines grows with scope complexity.The reported categories include Traversal, Position, and Condition; the growing gap is attributed to denser partial-violation signals from graded rewards.Source: Figure 3 · p. 8, §6.3 · p. 8
  • SCOPEINSTRUCT shows greater constraint diversity than three commonly used training datasets across Self-BLEU-4, Distinct-2, and Entropy-4.These metrics assess inter-constraint similarity, unique bigram proportion, and four-gram diversity and distributional balance.Source: Table 5 · p. 9, §6.3 · p. 9
  • Tool-Grounded Verification outperforms pure LLM judges and rule-based verifiers across judge models, and also performs best on IF-RewardBench.Generating tools without conditioning on the response causes a consistent performance drop.Source: §6.3 · pp. 8–9, Table 4 · p. 9
  • ScopeIF outperforms binary-reward baselines throughout training, with a more pronounced advantage on Qwen3-4B than on Qwen3-8B.The curves remain upward-trending at the final step and have not plateaued under the current training budget.Source: Figure 4 · p. 9, §6.3 · p. 9
  • The proportion of response pairs receiving identical rewards drops sharply as reward granularity increases from ILA to CLA and ScopeIF, and the gap persists during training.Identical rewards provide no relative preference between paired responses; the authors report that graded rewards substantially alleviate reward sparsity.Source: Figure 4 · p. 9, §6.3 · p. 10

7 CONCLUSION

ScopeIF combines a three-dimension constraint schema and a scope-aware instruction dataset with tool-grounded verification and graded reward modeling. Experiments show consistent gains over existing frameworks and reward paradigms, especially on complex scope-aware constraints.

7 CONCLUSION

  • ScopeIF combines tool-grounded verification with graded reward modeling to provide reliable, dense RL supervision for scope-aware instruction-following.Its unified schema factorizes constraints into Scope, Target, and Range and underlies SCOPEINSTRUCT, a dataset of scope-aware instructions.Source: §7 · p. 10
  • The framework’s experiments show consistent improvements over existing frameworks and reward modeling paradigms, particularly on complex scope-aware constraints.Source: §7 · p. 10
  • The schema is intended to support diverse constraint synthesis and computation of violation degrees.Source: §7 · p. 10

AI USE STATEMENT

The authors report using generative AI to generate synthetic datasets, clean and reformat data, and polish figures and the manuscript. The accompanying appendices define the constraint schema and list prompt templates used in the work.

AI USE STATEMENT

  • Generative AI tools were used for synthetic data generation, data cleaning and reformatting, and polishing figures and the manuscript; the authors reviewed the assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.1 SCOPE

  • Scope is instantiated by response units and a selector that isolates governed segments, with units ranging from entire responses to characters.Selectors include Global, Traversal, Position, and Condition operations, which can be recursively composed into nested scopes.Source: §A.1 · p. 15

A.2 TARGET

  • Target specifies the objects measured within a selected Scope and organizes 15 fine-grained subcategories into six major categories.The supplied examples include capacity, literals, structure, checklist, pattern, and relations.Source: §A.2 · pp. 15–16

A.3 RANGE

  • Range specifies admissible values for a Target through four absolute categories and two relative categories.The listed categories are one-sided bounds, prohibition, equality, intervals, comparison, and arithmetic relations; qualitative constraints can be represented as binary compliance with an Equality Range of one.Source: §A.3 · p. 16

B LIST OF PROMPT TEMPLATES

  • The appendices list prompt templates for atomic constraint generation, constraint crossover, quality assessment, target extraction, tool design, and tool-grounded judgment.Source: B LIST OF PROMPT TEMPLATES · pp. 16–17
  • Figures 5 and 6 present Scope and Range distributions, and the Target distribution, respectively, for constraints in SCOPEINSTRUCT.Source: Figure 5 · p. 17, Figure 6 · p. 17

C DATA STATISTICS OF SCOPEINSTRUCT

This appendix characterizes SCOPEINSTRUCT’s size and semantic diversity, then details range-deviation and format components used to compute training rewards.

C DATA STATISTICS OF SCOPEINSTRUCT

  • SCOPEINSTRUCT contains 17,968 instructions, split into 16,968 training and 1,000 test examples, with 1–12 constraints per instruction and a 4.83 average.Its Scope, Target, and Range category distributions are also reported.Source: C DATA STATISTICS OF SCOPEINSTRUCT · p. 17
  • SCOPEINSTRUCT has the highest within-cluster distance at every tested k, with slower decay that indicates more semantically dispersed constraints than baseline datasets.The comparison uses BGE-M32 embeddings and k-means; greater within-cluster distance indicates constraints are less reducible to a few prototypes.Source: C DATA STATISTICS OF SCOPEINSTRUCT · pp. 17–18, Figure 7 · p. 17

D DETAILS OF REWARD CALCULATION

  • For each measured object, the absolute deviation is zero inside its permissible interval and otherwise equals its distance to the nearest boundary.The deviation is computed as max(l_ij − a_ij, a_ij − u_ij, 0).Source: D DETAILS OF REWARD CALCULATION · p. 18
  • The final training reward multiplies the hierarchical reward by a binary format reward that is zero for truncated responses or unclosed <think> blocks.When a reasoning block is present, constraint verification uses only the final-answer text; the same format reward is applied to RL-ILA and RL-CLA.Source: D DETAILS OF REWARD CALCULATION · p. 18

E.1 EVALUATION METRICS

The evaluation covers four precise-instruction-following benchmarks with prompt- and instruction-level accuracy measures, alongside standard accuracy on three general benchmarks.

E.1 EVALUATION METRICS

  • IFEval and IFBench use prompt- and instruction-level accuracy under strict and loose evaluation, while IFHierBench reports only strict prompt- and instruction-level accuracy.Prompt accuracy requires all constraints to be satisfied; instruction accuracy measures the proportion satisfied, and loose evaluation normalizes responses before verification.Source: §E.1 · p. 18
  • On the authors’ test set, the reported metrics are CSR (Constraint Success Rate) and ISR (Instruction Success Rate).The passage identifies these metrics but does not provide their formulas here.Source: §E.1 · p. 18
  • MATH-500, GPQA, and MMLU-Pro are evaluated using standard answer accuracy.Source: §E.1 · p. 19

E.2 IMPLEMENTATION DETAILS

This section reports DPO and GRPO training settings, analyzes constraint-category performance, and describes verification procedures used for judge and rule-based methods.

E.2 IMPLEMENTATION DETAILS

  • DPO uses five sampled responses per instruction to form highest- versus lowest-reward preference pairs, while GRPO uses 32-sized train and PPO mini-batches.DPO trains for one epoch with batch size 128; GRPO runs for one epoch and selects a checkpoint using IFEval validation.Source: §E.2 · p. 19

E.3 DETAILED RESULTS OF EACH CONSTRAINT CATEGORY

  • Category results show that performance is lowest on recursively composed Nested scopes, while Capacity and Relations targets, Equality and Intervals ranges, and Arithmetic Relations are particularly difficult.Global scopes are generally easiest; One-Sided Bounds and Prohibition ranges are comparatively easier.Source: §E.3 · p. 20

E.4 DETAILS OF CONSTRAINT VERIFICATION METHODS

  • The pure LLM judge returns a direct judgment and per-object statistics, with the statistics taking precedence when they conflict; rule-based verification uses a generated tool for a binary judgment.Pairwise agreement compares win, tie, or lose labels derived from the number of constraints satisfied.Source: §E.4 · p. 21

F LIMITATIONS

The authors identify potential synthetic-data bias from reliance on one generation model and computational overhead from tool-grounded verification during training.

F LIMITATIONS

  • SCOPEINSTRUCT relies primarily on Seed-2.0-Pro for generation, which may introduce model-specific preferences; broader generation models and human auditing are proposed to improve robustness and diversity.GPT-OSS-120B is used for quality inspection as a mitigation.Source: F LIMITATIONS · p. 21
  • Tool-grounded verification adds computational overhead during reward calculation, but that cost is limited to training and does not affect inference efficiency.The authors identify lightweight judges, batched verification, and reusable tools as directions for reducing overhead.Source: F LIMITATIONS · p. 21
Loading 2609.32189v1…