PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim, Wonjun Kang, Albert No, Jaehyung Kim

arXiv:2610.19525v1cs.LGcs.CL

TL;DR

Verifiable instruction-following benchmarks measure output constraints but can overlook whether responses substantively answer requests, and verifier-based RL can reproduce that mismatch. CALIF calibrates verifier rewards using reference-policy likelihoods; it improves constraint satisfaction while preserving response quality and generalizes to rubric-based benchmarks.

  • Problem

    Verifiable instruction-following benchmarks check constraints but can give full credit without measuring whether responses substantively answer the request.

    Source: §1 · pp. 1–2

  • Method

    CALIF calibrates verifier rewards within a likelihood region defined by constrained and unconstrained responses from the pre-RL reference policy.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    CALIF averages 34.64 Constraint Satisfaction and 4.59 Response Quality on IFBench, and outperforms base policies in 40 of 45 rubric-benchmark model–metric comparisons.

    Source: §5.2 · pp. 6–7

  • Takeaways & Limitations

    Calibrating rather than maximizing reference likelihoods can mitigate reward hacking without additional models or explicit quality supervision.

    Source: §7 · p. 9

Abstract

from arXiv · show

Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.

1 INTRODUCTION

Verifier-based instruction-following benchmarks can reward constraint compliance without measuring whether responses answer the request. The paper diagnoses this gap and introduces CALIF to preserve response quality while improving constraint satisfaction.

1 INTRODUCTION

  • VIF verifiers check constraints, not whether a response answers the request, so full verifier credit can mask poor substantive answers.Frontier models with higher Constraint Satisfaction tend to have lower Response Quality.Source: §1 · pp. 1–2
  • In OLMo 3 DPO–RLVR pairs, RLVR checkpoints show higher Constraint Satisfaction but lower Response Quality, consistent with reward hacking.The verifier reward can reinforce response failures that VIF evaluation overlooks.Source: §1 · p. 2
  • CALIF calibrates verifier rewards within a prompt-specific likelihood region defined by constrained and unconstrained responses from the pre-RL policy.It reuses the reference policy maintained for KL regularization and requires at most one extra teacher-forced forward pass per rollout.Source: §1 · p. 2
  • Across five configurations, CALIF averages 34.64 Constraint Satisfaction and 4.59 Response Quality on IFBench, and beats baselines in 40 of 45 rubric-benchmark comparisons.IF-RLVR averages 44.93 Constraint Satisfaction and 2.96 Response Quality on IFBench.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work evaluates instruction following with both LLM-based judgments and programmatic checks, while reward-hacking mitigations add explicit feedback at additional training cost.

2.1 INSTRUCTION-FOLLOWING EVALUATION

  • LLM-judged benchmarks rate response quality or compare responses, while rubric-based work assesses individual requirements and their combinations.VIF benchmarks instead use programmatic checks of explicit constraints; IFEval covers length, keywords, and format, while IFBench covers a broader range.Source: §2.1 · p. 3

2.2 REWARD HACKING IN RLVR

  • Reward hacking describes proxy-reward optimization diverging from the intended objective, including RLVR models exploiting verification flaws without improving the task.Prior mitigations add learned preference reward models or LLM-based intent checks, which require an additional model or judge during training.Source: §2.2 · p. 3

3 THE EVALUATION BLIND SPOT IN VERIFIABLE INSTRUCTION FOLLOWING

The section examines whether verifier-based evaluation and training capture both constraint compliance and substantive answers to requests.

3 THE EVALUATION BLIND SPOT IN VERIFIABLE INSTRUCTION FOLLOWING

  • VIF benchmarks append output constraints to requests and use programmatic verifiers, but successful instruction following also requires a substantive answer.The paper notes that some model assessments use only IFBench constraint-compliance scores.Source: §3 · p. 4
  • Across models, higher Constraint Satisfaction tends to coincide with lower Response Quality, and adding constraints degrades answer quality, especially for models with the highest Constraint Satisfaction.A response that satisfies a constraint can count as successful under standard VIF even when it fails to answer the request adequately.Source: §3 · p. 4
  • All three matched OLMo 3 DPO–RLVR pairs show higher Constraint Satisfaction but lower Response Quality for RLVR checkpoints.These observations are consistent with reward hacking under IF-RLVR.Source: §3 · p. 4

4 CALIF: CALIBRATED LIKELIHOOD FOR INSTRUCTION FOLLOWING

The section uses Bayes’ rule to identify request-only response likelihood as a factor omitted by verifier rewards, then introduces CALIF to reward constraint satisfaction within a reference-calibrated likelihood region.

4 CALIF: CALIBRATED LIKELIHOOD FOR INSTRUCTION FOLLOWING

  • Verifier rewards favor constraint compliance but omit response likelihood under the request alone, so optimizing them can reduce request-answer quality.The paper frames the target response probability as combining constraint satisfaction and likelihood under the request.Source: §4 · pp. 4–5
  • Directly rewarding request-only likelihood can encourage predictable repetitions that inflate likelihood without improving response quality.Source: §4 · p. 5
  • CALIF defines a desired likelihood interval from constrained and unconstrained responses sampled from the frozen pre-RL reference policy.Both responses are scored under the request alone; the lower bound reflects reference request consistency, and the upper bound is a conservative ceiling.Source: §4 · p. 5
  • Within the interval, CALIF adds a fixed bonus to positive verifier rewards; below it, the verifier reward remains, and above it, reward is suppressed.When the reference bounds are reversed, the method retains verifier reward only at or below the upper bound.Source: §4 · p. 5
  • CALIF computes its reference responses and likelihoods before RL and uses the existing reference policy, adding no model and at most one extra forward pass per rollout.Source: §4 · p. 5

5 EXPERIMENTS

The experiments compare CALIF with IF-RLVR across model configurations and broader rubric-based benchmarks, then analyze constraint-type outcomes and LLM-judge rewards. CALIF generalizes beyond verifiable constraints and preserves response quality more effectively than verifier-only or judge-reward alternatives.

5.1 EXPERIMENTAL SETUP

  • The matched comparisons cover five model configurations trained on 95,373 prompts with one to five verifiable constraints, using GRPO and a shared rollout temperature and KL coefficient.CALIF uses α = 0.1 in the main experiments.Source: §5.1 · p. 6, §5 · p. 6

5.2 MAIN RESULTS

  • CALIF improves over base policies in 40 of 45 rubric-benchmark model–metric comparisons, versus 14 for IF-RLVR, and beats IF-RLVR in 41 of 45 comparisons.All four exceptions are on Format, the dimension most closely aligned with training constraints.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 ANALYSIS

  • On harder constraint types, especially word-level constraints, IF-RLVR’s largest constraint-satisfaction gains accompany its largest quality decline and shorter responses; CALIF preserves quality without that shortening.Figure 7 summarizes performance across constraint types, with bars and error bars showing means and standard deviations across five model settings.Source: Figure 7 · p. 7, §5.3 · p. 7
  • IF-RLVR’s Fail→Pass and Pass→Pass groups lose an average 2.05 and 1.26 Response Quality points, while CALIF’s corresponding changes are −0.28 and +0.01.Table 2 reports quality changes for verifier-passing outcomes, aggregated over five model configurations.Source: Table 2 · p. 7, §5.3 · p. 7

5.4 COMPARISON WITH LLM-JUDGE REWARDS

  • CALIF beats both LLM-judge reward configurations on 10 of 11 metrics for Tülu 3 8B and 7 of 11 for Qwen3-4B thinking mode.The judge baselines generally yield higher Response Quality, consistent with directly rewarding that metric using the evaluation prompt.Source: §5.4 · p. 8
  • CALIF reuses the reference policy and needs neither a separate judge model nor judge-score generation, while requiring at most one additional teacher-forced forward pass per rollout.It also avoids specifying explicit quality criteria for judge-based supervision.Source: §5.4 · p. 8

6 ABLATION STUDIES

The ablations show that directly rewarding reference likelihood can produce repetitive likelihood inflation, while CALIF’s bounded likelihood region addresses distinct failure modes. The bounds remain robust to sampling choices, and CALIF retains larger constraint-satisfaction gains than KL tuning alone at comparable response quality.

6.1 DIRECT LIKELIHOOD REWARDS ENCOURAGE LIKELIHOOD INFLATION

  • Direct likelihood reward raises reference log-likelihood from −1.50 to −0.68 while Response Quality falls from 3.98 to 1.91, with longer, more repetitive responses.Average response length increases from 485 to 1,677 tokens, and unique token 2-grams fall to 0.177 versus 0.713 for the base policy and 0.698 for CALIF.Source: §6.1 · p. 8

6.2 BOTH LIKELIHOOD BOUNDS ARE NECESSARY

  • Without the lower likelihood bound, Constraint Satisfaction reaches 43.00 but Response Quality falls to 2.16; without the upper bound, Response Quality falls to 3.51 despite increased reference log-likelihood.Only the full likelihood region improves both dimensions over the base policy; Table 4 compares the reward and region variants.Source: §6.2 · p. 8, Table 4 · p. 9

6.3 ROBUSTNESS TO LIKELIHOOD BOUND ESTIMATION

  • Both narrower and wider likelihood regions built from confidence-interval endpoints perform similarly to the default construction and improve Constraint Satisfaction and Response Quality over the base policy.The bounds are estimated from eight sampled reference responses per conditioning setting at temperature 1.0.Source: §6.3 · p. 9

6.4 CALIF GOES BEYOND KL REGULARIZATION

  • At β = 0.05, IF-RLVR matches base Response Quality at 3.98 but raises Constraint Satisfaction only from 22.22 to 23.00; CALIF at α = 0.1 reaches 28.78 and 4.01, respectively.Across tested α values, CALIF preserves base-policy quality while improving Constraint Satisfaction, and it remains effective at β = 0.Source: §6.4 · p. 9

7 CONCLUSION

The conclusion presents CALIF as a reference-calibrated likelihood-region approach that improves constraint satisfaction while preserving response quality, without additional models or explicit quality supervision. The appendix prompt rates responses on specified quality dimensions and returns an integer score.

7 CONCLUSION

  • Across five model configurations, CALIF improves constraint satisfaction while preserving response quality through a reference-calibrated likelihood region, with gains beyond training verifiers.The authors conclude that calibrating rather than maximizing reference likelihood can mitigate reward hacking without additional models or explicit quality supervision.Source: §7 · p. 9

A LLM-JUDGE PROMPT

  • The LLM-judge prompt requests an integer score from 1 to 10 based on helpfulness, relevance, accuracy, depth, creativity, and constraint conformance.The prompt’s required output is a JSON score field.Source: A LLM-JUDGE PROMPT · p. 12

B EXPERIMENTAL DETAILS

The experimental details specify the multi-constraint training data, model and GRPO settings, CALIF likelihood configuration, and evaluation sampling and rating procedure.

B EXPERIMENTAL DETAILS

  • Training uses 95,373 prompts formed by augmenting Tülu-3 SFT instructions with one to five automatically verifiable constraints from IFEval and IFTrain.Source: B EXPERIMENTAL DETAILS · p. 12
  • The study trains Qwen3-1.7B and Qwen3-4B in thinking and non-thinking modes and Tülu 3 8B DPO for four epochs, using GRPO on eight H100 GPUs.Qwen3-4B thinking mode runs for three epochs; training uses eight rollouts per prompt, global batch size 1,024, and KL coefficient 0.001.Source: B EXPERIMENTAL DETAILS · p. 12
  • CALIF uses anchor responses sampled at the rollout temperature and sets α = 0.1 in the main experiments.For thinking models, likelihoods include full reasoning traces; for non-thinking models, the reference-policy likelihood is computed over final-answer tokens only.Source: B EXPERIMENTAL DETAILS · p. 12
  • Evaluation uses provider-recommended sampling settings, maximum response lengths of 16,384 tokens in thinking mode and 2,048 in non-thinking mode, and three runs reporting mean and standard deviation.GPT-5.6 Sol rates Response Quality unless otherwise specified.Source: B EXPERIMENTAL DETAILS · p. 12

C RUBRIC-BASED INSTRUCTION-FOLLOWING BENCHMARKS

The section describes three rubric-based instruction-following benchmarks and their evaluation metrics, alongside captions identifying a pairwise comparison table and a KL-penalty figure.

C RUBRIC-BASED INSTRUCTION-FOLLOWING BENCHMARKS

  • AdvancedIF evaluates rubric-based instruction following across ComplexIF, Carried Context, and System Steerability; these experiments evaluate only single-turn, multi-requirement ComplexIF requests.The reported metrics are Overall Pass Rate and Micro Rubric Pass Rate.Source: C RUBRIC-BASED INSTRUCTION-FOLLOWING… · p. 12
  • Overall Pass Rate measures responses satisfying all rubric criteria, while Micro Rubric Pass Rate pools satisfied criteria across responses.Source: C RUBRIC-BASED INSTRUCTION-FOLLOWING… · p. 12
  • FollowBench covers five constraint categories and their combinations across five difficulty levels that progressively add constraints.Its metrics are Hard Satisfaction Rate, Soft Satisfaction Rate, and Consistent Satisfaction Levels.Source: C RUBRIC-BASED INSTRUCTION-FOLLOWING… · pp. 12–13
  • Table 5 presents CALIF-versus-IF-RLVR pairwise comparisons on IFBench, with counts from CALIF’s perspective and half credit for ties in adjusted win rates.Source: Table 5 · p. 13
  • Figure 9 compares CALIF with and without the KL penalty at α = 0.1; IF-RLVR uses β = 0.001.Source: Figure 9 · p. 13
  • FollowBench reports satisfaction of all constraints, the average fraction satisfied, and the average number of consecutive difficulty levels fully satisfied.Source: C RUBRIC-BASED INSTRUCTION-FOLLOWING… · p. 13
  • ComplexBench evaluates instructions combining constraints through conjunctions, conditional branches, and sequential dependencies.Its scoring-question accuracy covers Lexical, Semantic, Utility, and Format constraints.Source: C RUBRIC-BASED INSTRUCTION-FOLLOWING… · p. 13

D.1 PAIRWISE RESPONSE QUALITY EVALUATION

CALIF achieved a 78.10% adjusted win rate over IF-RLVR in pairwise response-quality evaluation. Without KL regularization, it still improved constraint satisfaction over the base policy while largely preserving response quality.

D.1 PAIRWISE RESPONSE QUALITY EVALUATION

  • 78.10% adjusted win rate: CALIF’s overall pairwise comparison with IF-RLVR on IFBench favored CALIF.A win or loss counted only when judgments agreed across both presentation orders; ties included order inconsistencies.Source: §D.1 · p. 13

D.2 CALIF WITHOUT KL REGULARIZATION

  • With β = 0, CALIF improved Constraint Satisfaction over the base policy while largely preserving Response Quality.Figure 9 compares this variant with default CALIF and IF-RLVR, both using β = 0.001.Source: §D.2 · p. 13
  • Removing the KL penalty leaves only the reference-policy forward pass conditioned on x for reward computation; per rollout, CALIF needs no more reference passes than KL-regularized IF-RLVR, excluding offline bound construction.Source: §D.2 · p. 13

D.3 A QUALITATIVE EXAMPLE OF LIKELIHOOD INFLATION

The qualitative examples include a direct likelihood-bonus response and comparisons of IFBench responses for musical definitions and code explanation.

D.3 A QUALITATIVE EXAMPLE OF LIKELIHOOD INFLATION

  • Table 6 compares Qwen3-1.7B base-policy responses with responses from IF-RLVR, CALIF, and a direct likelihood bonus.The direct likelihood-bonus response is labeled Response Quality 1/10 and Log-Likelihood -0.04.Source: Table 6 · p. 14, §D.3 · p. 14

D.4 ADDITIONAL QUALITATIVE EXAMPLES

  • Tables 7 and 8 provide qualitative comparisons of IFBench responses for musical definitions and code explanation, respectively.Source: Table 7 · p. 15, Table 8 · p. 15

Related papers

Loading 2610.19525v1…