PaperlayerView exact revision on arXiv

Source-linked AI summary

LIFT: Scoped Constraints for Evaluating and Training Instruction Following

Eran Goldstein, Yuval Belfer, Yonatan Belinkov, Daniel Gissin

arXiv:2610.28819v1cs.CL

TL;DR

Common IF benchmarks use readily checked constraints that may be detached from the requested task, while richer alternatives can require expert authoring and be difficult to extend. LIFT-Bench uses scoped constraints for document drafting, including content requirements, and pairs the benchmark with Lift-Train. Experiments report density-sensitive performance and less reward hacking with Lift-Train than with IFTrain.

  • Problem

    Many IF benchmarks use constraints independent of the main task, with limited diversity, human-authored data, restricted openness, or no control for constraint density.

    Source: §1 · p. 1

  • Method

    LIFT-Bench assigns verifiable and judged constraints to document sections, while Lift-Train uses the same task pipeline and benchmark scoring function as its RL reward.

    Source: §1 · p. 1, §5 · p. 7

  • Results

    Frontier-model scores degrade as constraint density increases, globally scoped constraints are hardest despite an order of magnitude fewer constraints, and IFTrain training often leads to reward hacking unlike Lift-Train training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The released benchmark’s modular design supports analyses across constraint density, mechanism, and type, while its training counterpart makes task adherence part of the reward.

    Source: §7 · p. 9

    The training comparison is caveated because LIFT-Bench topics are wiki-seeded drafting tasks, whereas IFTrain prompts come from WildChat.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

Common instruction following (IF) benchmarks, such as IFEval, evaluate the ability of language models (LMs) to follow simple constraints that can easily be verified programmatically. These same constraints, in turn, are used as reward signals for reinforcement learning. This design may lead to reward hacking of constraints, pushing models away from the main task. Recent attempts to address this resulted in benchmarks with richer constraints, but required expert labor, remained closed-access, and cannot be easily extended. We present Lift-Bench, an IF benchmark that asks a model to draft a document given a topic and a structured outline. Constraints are scoped to certain document sections, providing finer control over constraint density. Lift-Bench includes both verifiable and judged constraints, as well as a training counterpart, Lift-Train. Importantly, Lift-Bench provides both main task and instruction-following constraints, mitigating shallow reward hacking. On experiments with multiple models, we find that frontier models struggle with the harder subsets of our benchmark and analyze model performance along multiple axes. We also find that training on Lift-Train is much less susceptible to reward hacking than training on the common IFTrain. Lift-Bench and Lift-Train are publicly available, as well as the data generating pipeline.

1 INTRODUCTION

Existing IF benchmarks often separate constraints from the main task and offer limited constraint diversity, extensibility, and control over constraint density. LIFT-Bench scopes constraints to document elements and combines verifiable checks with judged content requirements; experiments reveal difficulty patterns and greater resistance to reward hacking from Lift-Train than from IFTrain.

1 INTRODUCTION

  • Existing benchmarks use constraints and checkers independent of the main task, have limited constraint diversity, rely on human-written constraints, and often lack open training data or extensibility.They also assess difficulty by total constraint count rather than constraint spread or density across a response.Source: §1 · p. 1
  • LIFT-Bench assigns constraints to elements in a topic-based document outline, evaluating each constraint only against its attached element.This distinguishes a requirement for a paragraph somewhere from one applying to a specified section.Source: §1 · p. 1
  • Verifiable checks assess deterministic constraints, while LM-judged constraints assess generated content requirements; together with synthesis and scoping, they make content adherence part of the reward and density configurable.The paper also releases an open-source synthesis pipeline and analyses enabled by its factored design.Source: §1 · p. 2
  • Frontier models’ scores degrade with increasing constraint density, globally scoped constraints are hardest despite an order of magnitude fewer constraints, and IFTrain training often leads to reward hacking while Lift-Train training does not degrade on AdvancedIF.The benchmark has 784 examples across 10 splits; Lift-Train has 34,258 examples.Source: §1 · p. 2

2 RELATED WORK

Prior IF benchmarks either emphasize globally checked, programmatically verifiable constraints or use richer, often human-authored checklists. LIFT-Bench is presented as combining content and scoped constraints with a released training set and synthesis pipeline.

2 RELATED WORK

  • IFEval and IFBench check constraints against the whole response, and their metrics do not reflect main-task adherence; richer checklist benchmarks add content requirements but are hand-authored and frozen.IFBench publishes IFTrain, while the cited checklist benchmarks provide neither a training set nor synthesis code.Source: §2 · p. 2
  • AdvancedIF and ComplexConstraints use expert-written rubrics, while their training data are unreleased; AdvancedIF also keeps its generator, verifier, and RL code closed.Both report rubric-reward transfer to held-out IF benchmarks.Source: §2 · p. 2
  • MOSAIC identifies the gap between constraint compliance and task success, but its constraints are globally scoped and limited to format, style, or structure.LIFT-Bench instead scores task adherence and scopes constraints to sections.Source: §2 · pp. 2–3
  • LIFT-Bench is described as the only known IF benchmark with content constraints, scoped constraints, and a released training set and pipeline.Table 1 compares related IF benchmarks; a checkmark denotes an existing but unreleased feature.Source: §2 · p. 3, Table 1 · p. 3

3 LIFT-BENCH

LIFT-Bench represents document-drafting tasks with outlines and section-scoped constraints, scores structure and content in a shared metric, and synthesizes tasks through a staged pipeline.

3.1 TASK AND CONSTRAINT MODEL

  • A scoped constraint attaches to one outline leaf and is evaluated only against that section’s text, enabling fine-grained analysis; flat tasks instead apply constraints globally.Density is the number of constraints attached to a section and is set during generation; for flat tasks, density equals total constraint count, so the two regimes are not on a common scale.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 SCORING

  • Task scores are weighted sums of binary per-constraint results, with structure and body weights configurable; the released configuration assigns 10% to structure and 90% to content.The same scalar is used as both the benchmark metric and RL reward; because content constraints dominate, formatting compliance without a meaningful document scores close to zero.Source: §3.2 · pp. 3–4

3.3 SYNTHESIS PIPELINE

  • The synthesis pipeline samples Wikipedia topics, constructs and expands outlines, then uses parallel per-type generators to propose candidate constraints over the outline.The pipeline is presented as a sequence of atomic stages.Source: §3.3 · p. 4, Figure 2 · p. 4

4. Task configuration

Task configuration varies constraint sampling and reward allocation, while the released splits are designed to compare constraint loads within shared scoping regimes.

4. Task configuration

  • Constraint sampling controls how many constraints each section receives, how many sections receive any, and which constraint types appear; reward shaping sets the structure/content budget split.Tasks can be rendered as structured prompts with per-header constraints or flat prompts applying the same constraints globally.Source: §4 · p. 5

3.4 THE RELEASED SPLITS

  • The ten splits come from short-outline, long-outline, and flat runs; within each run, topics and outlines are shared while constraint counts vary.Splits are comparable only within the same scoping regime, and the ten splits do not form a monotonic difficulty scale.Source: §3.4 · p. 5

4 BENCHMARK EVALUATION EXPERIMENTS

Across 17 models, structured splits approach ceiling for the strongest models, but flat splits remain unsaturated and rankings change across scopes. Performance analyses show that constraint load, remote constraints, and constraint type distinguish model behavior.

4.1 SETUP

  • The evaluation covers 17 models on 10 splits, with each example generated eight times and scored by its mean across runs.Intervals are 95% confidence intervals across examples; Gemma-4-31b scores judged constraints.Source: §4.1 · p. 5

4.2 OVERALL RESULTS

  • GPT-5.5 averages 0.96 on structured splits, but no model exceeds 0.85 on flat splits and weaker models reach 0.47 on the densest structured splits.Table 2 reports mean scores per split for selected models; Appendix F gives all 17 models.Source: §4.2 · p. 5, Table 2 · p. 5
  • Kimi K3 and DeepSeek V4 Pro have the highest overall averages, matching the strongest closed models on structured splits and surpassing them on flat ones.GPT-5.5 ranks first on structured splits but fifth on flat3, while Qwen3.8 rises from seventh to third.Source: §4.2 · pp. 5–6

4.3 WHERE MODELS FAIL: LOAD, MECHANISM, AND TYPE

  • For GPT-5 nano, judged constraints affect performance more than verifiable constraints, while GPT-5.5 declines mildly across the constraint-load plane.At the densest loads, GPT-5 nano scores 0.2–0.5 while GPT-5.5 stays above 0.7.Source: §4.3 · p. 6, Figure 3 · p. 6
  • At fixed per-section density, adding constraints elsewhere has little effect on frontier models but degrades smaller models, including GPT-5 nano by −5.5%.GPT-5.5 declines −1.5% per density step; the Table 3 comparison estimates section-score change as document load increases.Source: §4.3 · p. 6, Table 3 · p. 6
  • Constraint-type performance varies by model: frontier models are near ceiling on document structure, emoji count, and content, but vary most on artifacts and case type.On code snippets, Kimi K3 scores 0.98 against 0.73–0.83 for the other three frontier models.Source: §4.3 · p. 7, Figure 4 · p. 7

5 LIFT-TRAIN

LIFT-Train uses the benchmark’s task and scoring design as an RL training reward, with content constraints forming most of that reward. The experiments vary data composition and curriculum and compare against lexical, verifiable-only IFTrain.

5 LIFT-TRAIN

  • LIFT-Train uses the benchmark scoring function unchanged as the RL reward, so satisfying formatting constraints without drafting a meaningful document earns almost nothing.Most of the reward consists of content constraints.Source: §5 · p. 7
  • Unlike IFTrain’s lexical, verifiable-only reward, LIFT-Train scores task adherence through constraints within the same reward rather than a separate overall-quality signal.The authors hypothesize this makes LIFT-Train less prone to reward hacking and compare training on both datasets.Source: §5 · p. 7

5.1 TRAINING MIXTURES AND SETUP

  • Training mixtures vary data composition across structured-only, flat-only, combined, and combined-plus-IFBench constraints, and vary curriculum between fixed difficulty and easy-to-hard.Combined data is controlled for size; flat-only is the closer IFTrain comparison because it matches prompt format and constraint style but includes a content component.Source: §5.1 · pp. 7–8
  • The study uses GRPO on Qwen2.5 7B, Qwen2.5 32B, and reasoning Qwen3-32B for 180 steps, extending the Qwen2.5 runs to step 390.The longer runs probe for long-training pathologies.Source: §5.1 · p. 8

6 TRAINING EXPERIMENTS

LIFT-Train performs well on LIFT-Bench and, unlike IFTrain, also improves external judged benchmarks; training on IFTrain shows reward-hacking behavior. Mixture comparisons favor combining flat and structured training for performance across both split types, while easy-to-hard ordering makes no difference.

6.1 MAIN TRAINING RESULTS

  • Only LIFT-Train also improves external judged benchmarks, while IFTrain degrades performance on AdvancedIF and ComplexBench.Each recipe gains most on the benchmark its reward can see.Source: Table 4 · p. 8, §6.1 · p. 8
  • By step 180, IFTrain reward reaches 1.0 as median visible length falls from 114 to 28.5 words; by step 390, it falls to 18 words.Observed rollouts include bare constraint tokens, empty section headers, and phrase repetitions; the same base model trained on LIFT-Train does not show this high reward-hacking tendency.Source: §6.1 · p. 8
  • A flat-only LIFT-Train control produces healthy training without the length pathologies, though its topics still differ from IFTrain’s.The control matches IFTrain’s flat prompt format and IFEval-style constraints but includes content adherence in its reward.Source: §6.1 · p. 8

6.2 MIXTURE COMPOSITION AND TRAINING ORDER

  • The combined mixture reaches 0.896 on flat splits and 0.919 on structured splits, while single-scoping mixtures fall short on the other split type.Fixed-all structured-only training reaches 0.805 on flat splits; flat-only training reaches 0.798 on structured splits (base: 0.604 flat, 0.691 structured).Source: §6.2 · p. 9, Figure 5 · p. 9
  • Easy-to-hard structured curriculum scores on par with its fixed-difficulty counterpart.The authors suggest adjacent split overlap may limit the added diversity of ordering and leave deeper curriculum exploration to future work.Source: §6.2 · p. 9, Figure 5 · p. 9

7 CONCLUSION

LIFT-Bench scopes constraints to document sections and scores content requirements alongside formatting constraints, making constraint density controllable and task adherence part of the reward. The authors report that LIFT-Train is less susceptible to reward hacking than IFTrain and identify extensions to strengthen judging and broaden task coverage.

7 CONCLUSION

  • LIFT-Bench scopes constraints to sections and scores content requirements alongside formatting, making density controllable and task adherence part of the reward.Its modular design supports analyses across constraint density, mechanism, and type.Source: §7 · p. 9
  • LIFT-Train is less susceptible to reward hacking than IFTrain, which the authors attribute to its content-adhering reward component.Source: §7 · p. 9
  • The authors identify a finetuned rubric judge, richer constraint compositions, conditional and chained dependencies, and multiturn editing as possible extensions.The judge would harden the scoring stage used by both the benchmark and its reward.Source: §7 · p. 9
  • LIFT-Bench, LIFT-Train, and the synthesis pipeline are publicly available.Source: §7 · p. 9

A SYNTHESIS PIPELINE: DETAILED DESCRIPTION

The synthesis pipeline builds drafting tasks from topics and structured outlines, attaches constraints to sections, samples and scores them, and renders them as prompts. Its judge-scoring process evaluates judged constraints against the text for their specific scope.

A SYNTHESIS PIPELINE: DETAILED DESCRIPTION

  • The pipeline uses independent blocks with caching, so changing a late stage reuses earlier artifacts and density comparisons share documents and candidate constraints.Items that fail a stage are diverted and dropped; a block aborts if fewer than 20% survive.Source: A SYNTHESIS PIPELINE: DETAILED… · p. 13
  • Wikipedia Vital Articles titles seed topics, which the pipeline uses as free text; the source can be swapped without changing the rest of the pipeline.Structured tasks use pseudorandom outline skeletons: short documents average 5 headers and 4 leaves, while long ones average 11 headers and 9 leaves; flat tasks reduce to one scope.Source: A SYNTHESIS PIPELINE: DETAILED… · p. 13
  • Each constraint type proposes candidates independently, but conflicts are not resolved globally, so each type must avoid collisions during generation.Candidate pools are sampled later; Table 5 lists constraint types and candidate sources.Source: A SYNTHESIS PIPELINE: DETAILED… · pp. 13–14, Table 5 · p. 14
  • Sampling runs once per constraint family, drawing up to N constraints per section and limiting each pass to at most M leaf sections.The two passes select sections independently, making per-section density a settable quantity; released configurations vary M and N.Source: A SYNTHESIS PIPELINE: DETAILED… · p. 14, Table 6 · p. 15
  • Structured tasks add exact-match title and paragraph-existence constraints; scoring combines structure and content means with content weighted at 0.9.Flat tasks omit structure constraints and score as the mean of content constraints.Source: A SYNTHESIS PIPELINE: DETAILED… · p. 14
  • Prompts use either the structured default or an IFEval-style flat rendering; the attempted LLM verbalizer is unreleased after sometimes treating constraints as instructions to itself.Train/test assignment uses stable example IDs, while failed generated items are dropped; test retention falls from 90% for flat topics to 66% for long outlines.Source: A SYNTHESIS PIPELINE: DETAILED… · pp. 14–15

B JUDGE SCORING

  • Verifiable constraints return deterministic booleans, while judged constraints are scored by a light critic on a 1–5 scale and thresholded at 4.The judge is Gemma-4-31B, run once at temperature 0.6; the threshold is not calibrated against labeled judgments.Source: B JUDGE SCORING · pp. 15, 17
  • For each judged constraint, the judge receives a synthetic single-section request, the criterion, and only the text for that scope.This lets it assess the relevant passage directly rather than locate it within a potentially long document.Source: B JUDGE SCORING · p. 17

C JUDGE META-EVALUATION

The meta-evaluation compares scoped and unscoped judging against human labels and finds a substantial advantage for scoping across judge models. Additional comparisons find model choice matters less, while two tested prompt and scope variations have little effect.

C JUDGE META-EVALUATION

  • The annotated set contains 305 judged-constraint response pairs, sampled across judged types and the baseline judge’s verdict.Annotators label each pair using the same scope and criterion seen by the judge.Source: C JUDGE META-EVALUATION · p. 17
  • Scoping is worth roughly 11–19 macro-F1 points depending on the judge, and every scoped judge beats every unscoped one.The study evaluates 305 pairs: 227 valid and 78 invalid; whole-document judges mostly accept responses, while scoping makes them discriminative.Source: C JUDGE META-EVALUATION · pp. 17–18, Table 9 · p. 18
  • Alternative judge models add 4–17 points within the scoped setting, a smaller and less reliable effect than scoping.Gemma-4-31B is adopted as the strongest alternative at comparable serving cost.Source: C JUDGE META-EVALUATION · p. 18
  • Changing the scope level or adding the grading rubric moves macro-F1 by at most one point; the authors say the remaining gap to human agreement is unlikely to close by prompting alone.They identify a trained judge as a more promising route and threshold calibration as a cheaper first step.Source: C JUDGE META-EVALUATION · p. 18

D STATISTICS FOR RELEASED SPLITS

E EVALUATED MODELS

  • The appendix lists 17 evaluated models, including closed- and open-weight models.The main text names nine models in its Table 2.Source: E EVALUATED MODELS · p. 18

F FULL RESULTS

Across the evaluated models, globally scoped flat tasks are the hardest regime despite having far fewer constraints. The wider model set shows that most performance variation comes from smaller models, while section-load analysis measures how added constraints elsewhere affect a section’s score.

F FULL RESULTS

  • Table 10 reports mean scores per split for 17 models, with closed- and open-weight models in separate panels and structured and flat averages weighted equally in “avg (all)”.Bold marks the best value across all 17 models; averages include 95% confidence intervals across examples.Source: F FULL RESULTS · p. 18, Table 10 · p. 20

G GLOBALLY-SCOPED PROMPTS ARE THE HARDEST REGIME

  • At a matched budget of ≤10 constraints, all but Qwen3.8-A95B score significantly lower on flat than structured tasks; leading closed models score 0.65–0.74 on flat tasks.Flat tasks carry an order of magnitude fewer constraints, and Figure 7 compares mean scores with 95% confidence intervals.Source: G GLOBALLY-SCOPED PROMPTS ARE THE… · p. 21, Figure 7 · p. 21

H FULL MODEL SET

  • Frontier models are largely indistinguishable across the wider set, with score variation coming from smaller models.The appendix covers nine evaluated models plus eight smaller models from earlier rounds.Source: H FULL MODEL SET · p. 21

I SECTION-LOAD FITTING

  • Section-load slopes estimate how section scores change as constraints are added elsewhere while that section’s own constraint count stays fixed.Slopes are ordinary least-squares fits over density tiers for structured splits; short- and long-document settings are fitted separately, with no fit for short documents at the denser tiers.Source: I SECTION-LOAD FITTING · p. 21, Table 11 · p. 22

J REWARD HACKING ANALYSIS

Training on IFTrain produces short, degenerate outputs that satisfy lexical constraints without adhering to prompt content. Held-out completions receive full reward even when they do not answer the prompt.

J REWARD HACKING ANALYSIS

  • Training on IFTrain leads to short, degenerate outputs that optimize constraint scores without adhering to prompt content.Figure 9 tracks held-out validation pass rates every 10 steps using 200 completions per point; its caption identifies lexical-only reward as the hacked reward.Source: J REWARD HACKING ANALYSIS · p. 23, Figure 9 · p. 23

J.1 REWARD-HACKED COMPLETIONS

  • Every shown IFTrain completion earns full reward because its lexical checker does not inspect whether the response answers the question.The checker returns 1.0 when every constraint is satisfied and 1/256 otherwise; the reproduced completions come from held-out validation passes.Source: §J.1 · p. 23

Related papers

Loading 2610.28819v1…